Results 1 to 14 of 14
  1. #1
    junker is offline Novice
    Windows XP Access 2007
    Join Date
    Oct 2011
    Posts
    6

    query to extract data from million rows of data

    Guys - this is my first post... so, please be easy on me...



    I have attached table running into 2.5 million lines and need to capture rows with relevant data like country, county and city. Ideally same row in different table...

    for example;
    IF table1 has Field1 = type and Field2 = name, then copy field1, field2 and field6 into another table, say table2-row1 (that will fill 3 columns in row 1 of table 2)

    Now, search next few rows until you find Field1 = type and Field2 = region, then copy Field6 to table2-row1-col4 (so it does not override copied data. I think if I don't specify to put the value in row1-col4, it may replace the entire row with the value coming from second line of code...

    Then search next few rows until you find Field1 = type and Field2 = city, then copy Field6 to table2-row1-col5 (again to not override copied data earlier)

    Once you find a row with Field1 = type and Field2 = name AGAIN, close the first query above and start the loop again i.e.

    IF table1 has field1, field2 and field6 into another table, say table2-row1 (that will fill 3 columns in row 1 of table 2) and so on....

    The relevant data is only in few hundred rows out of 2.5 mil and therefore any other way seems difficult to extract the information like get these rows individually and then try to combine, seems more work.

    Let me know if something is not clear and any thoughts will be appreciated.

    Regards,
    A

    HTML Code:
    Field1        Field2        Field3        Field4        Field5        Field6
    type        country        data        type        data        USA
    source        name        type        String        >        ?
    tuy        tuy        tr        76        98        9
    type        region        oiu        oi        hu        PA
    dsa        county        oiu        oi        oi        philadelphia
    ads        city        iu        iu        nmj        philadelphia
    fgfh        hjk        gu        kv        khjg        k
    jkKl        iu        oiu        y        iuy        iu
    mnb        iu        yiu        iuy        iuy        SD
    type        country        data        type        data        USA
    source        name        type        String        >        ?
    tuy        tuy        tr        76        98        9
    type        region        oiu        oi        hu        NY
    dsa        county        oiu        oi        oi        NY
    ads        city        iu        iu        nmj        Manhattan
    fgfh        hjk        gu        kv        khjg        k
    jkKl        iu        oiu        y        iuy        iu
    mnb        iu        yiu        iuy        iuy        SD
    type        country        data        type        data        USA
    source        name        type        String        >        ?
    tuy        tuy        tr        76        98        9
    type        region        oiu        oi        hu        CA
    dsa        county        oiu        oi        oi        Marin
    ads        city        iu        iu        nmj        San Rafael
    fgfh        hjk        gu        kv        khjg        k
    jkKl        iu        oiu        y        iuy        iu
    mnb        iu        yiu        iuy        iuy        SD
    Last edited by junker; 10-20-2011 at 08:02 AM. Reason: CLARIFY BETTER

  2. #2
    Robeen is offline VIP
    Windows XP Access 2010 32bit
    Join Date
    Mar 2011
    Location
    Tulsa, Oklahoma.
    Posts
    1,596
    Can you do one run through Table1 and insert all the data that meets this criteria:
    Code:
    IF table1 has Field1 = type and Field2 = name, then copy field1, field2 and field6 into another table, say table2-row1 (that will fill 3 columns in row 1 of table 2)
    . . . into Table2?

    And then do a separate run through Table1 and insert field6 into the two columns of Table 10?

    Or does it all HAVE to be done in one pass?

    What do you mean by:
    so it does not override my first line of code
    I think what you want to do is not difficult.
    It might take a little while to execute because Table1 is so large, but it can certainly be done!
    Last edited by Robeen; 10-20-2011 at 07:26 AM. Reason: Typo.

  3. #3
    junker is offline Novice
    Windows XP Access 2007
    Join Date
    Oct 2011
    Posts
    6
    Hi Robeen,

    Sorry, it was typo. I did not realize I had table2 and then table10. all should be table2. I have updated the original query..

    Override is only for col4, if I don't specify to put the value in row1-col4, it may replace the entire row with the value coming from second line of code...

    Does it make sense?

  4. #4
    orange's Avatar
    orange is offline Moderator
    Windows XP Access 2003
    Join Date
    Sep 2009
    Location
    Ottawa, Ontario, Canada; West Palm Beach FL
    Posts
    16,850
    What exactly are you trying to do?
    What does the data represent/mean to you or your business?
    I see you have 2 posts - How much experience do you have with Access? Do you do any table/database design?
    If you can map out what you are trying to do, it might be easier for the rest of understand and offer assistance.

  5. #5
    junker is offline Novice
    Windows XP Access 2007
    Join Date
    Oct 2011
    Posts
    6
    I am trying to extract, in specific order, country region city information for all countries in the world. If you refer to sample table in my first post, it will give you an idea on the quality of data.

    It is for my research purposes. My experience with Access is limited and do not have much understanding of establishing queries in vba. I probably can try in excel (proficient) but it has limit of 1 mil rows and trying to find a easy way to do it.

    Let me know if it helps.

    Not sure if I understood your comment on two posts. This is my first post and second is the reply to the question Robeen asked...

  6. #6
    Robeen is offline VIP
    Windows XP Access 2010 32bit
    Join Date
    Mar 2011
    Location
    Tulsa, Oklahoma.
    Posts
    1,596
    Hi Junker,

    So - if I understand correct, you want to:

    1. Read through Table1 rows till you find a row that has:
    Field1 = type & Field2 = name

    2. Store Field1, Field2 & Field6 [to eventually copy it to table2-col1, col2, col3].

    3. Search following rows in Table1 till you find a row that has:
    Field1 = type and Field2 = region

    4. Store Field6 [to eventually copy it to table2-col4].

    5. Search following rows in Table1 till you find a row that has:
    Field1 = type and Field2 = city

    6. Store Field6 [to eventually copy it to table2-col5].

    7. Search following rows till you find another row with:
    Field1 = type & Field2 = name

    8. Write stored data into cols 1 - 5 into one row of Table2.

    9. Start process for building data for next row in Table2.

    Is that what you need?

  7. #7
    orange's Avatar
    orange is offline Moderator
    Windows XP Access 2003
    Join Date
    Sep 2009
    Location
    Ottawa, Ontario, Canada; West Palm Beach FL
    Posts
    16,850
    Is this an exercise of some sort?

    Are you familiar with these sites?
    http://en.wikipedia.org/wiki/ISO_3166-1_numeric
    http://unstats.un.org/unsd/methods/m49/m49regin.htm

  8. #8
    junker is offline Novice
    Windows XP Access 2007
    Join Date
    Oct 2011
    Posts
    6
    Absolutely correct!

    Thanks.

  9. #9
    Robeen is offline VIP
    Windows XP Access 2010 32bit
    Join Date
    Mar 2011
    Location
    Tulsa, Oklahoma.
    Posts
    1,596
    Hi Junker,

    I'm not sure if you said 'Absolutely correct!' to Orange's post or to mine.
    I'll stay out of the picture if it looks like Orange has you covered here!

    All the best!

  10. #10
    junker is offline Novice
    Windows XP Access 2007
    Join Date
    Oct 2011
    Posts
    6
    Hi Robeen,

    Apologies, it was my mistake that I did not realize there were two messages while posting. My post was reply to your msg and that is exactly I trying to do with the information.

    The raw information and various websites will provide same info as I have in my file. Sorry Orange, nothing against you - peace!

    Can you please help me build query in access?

    Regards,
    A


    Quote Originally Posted by Robeen View Post
    Hi Junker,

    I'm not sure if you said 'Absolutely correct!' to Orange's post or to mine.
    I'll stay out of the picture if it looks like Orange has you covered here!

    All the best!

  11. #11
    orange's Avatar
    orange is offline Moderator
    Windows XP Access 2003
    Join Date
    Sep 2009
    Location
    Ottawa, Ontario, Canada; West Palm Beach FL
    Posts
    16,850
    No problem from my end. I wasn't sure how to interpret the Absolutely correct! either.
    I wasn't sure if you just wanted an authoritative source for Country and Regions, or if you were learning to do something in Access. It wasn't clear to whom the response was directed.

    In any event it appears you have things under control.
    Good luck with your project.

  12. #12
    Stingaway is offline Efficiency Junkie
    Windows 7 64bit Access 2010 64bit
    Join Date
    Sep 2011
    Location
    The deep south. Keep going until you hit water basically.
    Posts
    224
    Is it just me, or does querying a million records in Access cause anyone concern here? Usually when I get up above 400,000 records (average data - not heavy workhorse data with 250 fields, etc...), Access starts to go a bit wonky.

    If I throw those records into SQL (insert the thought that a massive record scratch just happened, everyone is now shocked and staring and preparing the rotting vegetation to throw, but I had to ask...) and use Access for my front end, all is well.

  13. #13
    Robeen is offline VIP
    Windows XP Access 2010 32bit
    Join Date
    Mar 2011
    Location
    Tulsa, Oklahoma.
    Posts
    1,596
    Hi Junker,

    I have done this kind of thing but the way I do it is not as simple as just writing a query.

    I use a Function to do processing like what you need.

    I'll give you a Code outline of how I think I might approach it.

    First - a simple, non-code, outline:
    1. Create local variables to hold data from each of the fields you are searching for.
    2. Create a Recordset object [this will read one line of Table1 at a time].
    3. Set up a Do while Loop to go through each row of your Table1 data.
    3a. You might need to set up an inner Loop to handle each row of data that you will be writing.
    This inner loop will start when you find: Field1 = type and Field2 = name
    and end when you find: Field1 = type and Field2 = name AGAIN.
    When you DO find Field1 = type and Field2 = name AGAIN: you will write a row of data from your stored variables into Table2.
    4. Get the different fields from the row into your variables.
    5. Test your variables to see if they are Field1 = type and Field2 = name . . . etc.
    6. If the variables meet your criteria - hold on to them.
    7. Loop through rows in your recordset till you find Field1 = type and Field2 = name AGAIN.
    8. Write a row of data to Table2.

    Here's a rough outline of code that you can use.
    This is not tested code just a starting point.
    You'll have to play with the logic to get it right.

    Code:
    Function GetData()
     
    Dim db As DAO.Database
    Dim rs As DAO.Recordset
     
    Dim strtemp1, strtemp2 As String
    Dim str1, str2, str3, str4, str5, strSQL As String
     
    'Integer variable to tell you if the row to be written is complete.
    'This will be 2 when you hit the second Field1 = type and Field2 = name.
    Dim RowComplete as Integer 
    RowComplete = 0   
     
    On Error GoTo Err_Routine
     
     
    Set db = CurrentDb
    Set rs = db.OpenRecordset(strSQL, dbOpenSnapshot)
    strSQL = "Select * From [Table1]"
     
    With rs
     
    Do While Not rs.EOF
     
    'Get Field1 & Field2 data into your variables:
    strtemp1 = rs![Field1]
    strtemp2 = rs![Field2]
     
    'Check Field1 & Field2 values.
    If strtemp1 = "type" Then
        If strtemp2 = "name" Then
        RowComplete = RowComplete + 1
     
        If RowComplete = 2 Then    'The row is ready to be written.
     
    'Use something like this to write your data to Table2:
    StrSQL = "INSERT INTO Table2 (Field1, Field2, Field3, Field4, Field5) "
    StrSQL = StrSQL & "VALUES (" & "'" & str1 & "'" & ", " & "'" & str2& "'" & ", " & "'" & str3 & "'" & ", " & "'" & str4 & "'" & ", " & "'" & str5 & "'" & ", "  "); "
     
    'MsgBox StrSQL     'Use to test your SQL Statement.
     
    DoCmd.RunSQL StrSQL             'Insert the Data into Table2.
     
    'Reset all variables to 0 or "" for next 
    RowComplete = 0
    Str1 = ""
    Str2 = ""
    Str3 = ""
    Str4 = ""
    Str5 = ""
     
        Else                                'Row is NOT ready to be written.
            'Get first 3 fields for Table2 into variables.
            str1 = strtemp1
            str2 = strtemp2
            str3 = rs![Field6]  
        Else If strtemp2 = "region" Then
            str4 = rs![Field6]  
        Else If strtemp2 = "city" Then
            str5 = rs![Field6]     
     
        End If
        End If
    End If
     
    Loop
    End With
     
    Err_Routine:
    Exit Function
     
    End Function
    I hope this helps to get you started.
    I'm sorry I do not have time to get it all tested etc - but I'll help if you get stuck.

    You will have to iron out the logic in my 'building blocks'.


    Let me know if you run into trouble. You can start out by stripping the If Then Else part out - to test you can get your variables in and insert them.

    Make a copy of Table1 with just a few rows of data [15 - 20] to test first.

    All the best!

  14. #14
    junker is offline Novice
    Windows XP Access 2007
    Join Date
    Oct 2011
    Posts
    6
    Thanks Robeen,

    I'll work on it and let you know how it goes... Thanks once again for the help.

    Regards,
    A

    Quote Originally Posted by Robeen View Post
    Hi Junker,

    I have done this kind of thing but the way I do it is not as simple as just writing a query.
    All the best!

Please reply to this thread with any new information or opinions.

Similar Threads

  1. Extract/Populate Data
    By tylerg11 in forum Forms
    Replies: 1
    Last Post: 09-26-2011, 05:54 PM
  2. Extract data at end of cell
    By madsc1 in forum Access
    Replies: 2
    Last Post: 03-21-2011, 04:12 PM
  3. Extract Partial Data
    By madsc1 in forum Access
    Replies: 6
    Last Post: 03-16-2011, 03:43 PM
  4. Extract Text Data
    By tmcrouse in forum Queries
    Replies: 5
    Last Post: 05-25-2010, 11:34 AM
  5. Replies: 3
    Last Post: 05-13-2010, 08:18 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Other Forums: Microsoft Office Forums