Help on modiying a File Searcher!

[orange]

Do you have any ideas of how others (organizations similar to yours with similar needs) do this task(s). My guess is you need something to automate the parsing of articles, extraction of meaningful words and phrases and assignment of attributes. I would also think there is a customized set of terms, related terms, broader terms, narrower terms (thesaurus capability) at the heart of things.


You are almost looking for a "special search engine" ( a google/yahoo type tool) with a database with a lot of horsepower (SQL server, mySQL, other ).
I don't think this is an Access application if you look to growing at 600 articles per day, and better retrieval.
I recommend you investigate how others(that are doing it well/profitably) are doing things. Look specifically in your own processes where things are bogged down, and look for automation opportunities. Perhaps a packaged solution is available???
Does management recognize that there is an issue?
Have you considered a consultant to do some analysis and proposals to assist decision making?

[Fipper_SG]

I personnally believe the code from David Fenton could be used to solve my problem. It already scan through files, so all I need is to make some modifications to it...

[Fipper_SG]

I think I found a solution! (See attached database)

Instead of doing a centralized database, I'm thinking of doing personnal database where everyone totally manage their section.

The sample is working. However when I select a folder that has lots of files, access crashes... Is it because I'm asking access too much or there's a problem somewhere?

[orange]

Fipper_SG,
Just saw this thread again. Did you find a final solution. I tried your database using D Fenton's FileSearch and it works fine. It failed on me while calculating but it was looking at many jpgs.
When I use a more reasonable number of files it works fine and allows me to open the files found.
If you have anything further, please post.

[Fipper_SG]

Hi there,

I found a final solution but it's not entirely with access... I don't know why, but D Fenton's FileSearch doesn't seem to search through all pdf files... As I was not very familiar with the code, I took another direction.

My final is attached to this message.

Here's how I use it:

1- I use the Add Files Button to list the Files from a specific directory (a popup window appears to indicate the direcory).

2- I use DT Search to search through the files* (http://www.dtsearch.com/) and save the Search Result List as an XML File to a specific folder. The name I give to the XML File is the word that will be added as a keyword to the log. I do this for every keyword I wish to add.

*To search through the files, you first need to build an index. I make sure that this index search through the same directory that is listed in step #1

3- I use the Add Keywords button to add the keywords to the log (I need to indicate the location of the XML Files in the popup window)

If you know of any alternative that is faster than step #2, I would be very interested to hear about it!

[orange]

Glad you have a workable solution. Can you post a copy with a few records? I'll look at it later tonight or tomorrow.

[Fipper_SG]

Sure! Here it is.

[orange]

I've read through some of the dtSearch material on the web site and the Powerpoint info from the Canadian distributor's site.

Can you use dtSearch for all of your activity? Seems to me and I don't know the details and have never worked with dtSearch, that dtSearch will index your files. It seems that you could have dtSearch look for a number of word and synonyms and even include Noise Words to ignore. dtSearch would build its indexes for these keywords and allow you to search for combinations of keywords. It also appears you can request to view the various "found" documents and highlight the keywords.

If this is true (that's really a question not a fact necessarily), what part do you have to do in Access?

The documentation/slide presentation indicates you can have dtSearch on a PC or Network, so couldn't "your teams" use dtSearch for indexing and searching and retrieval?

I'm trying to get a broader picture of the processes involved to understand where Access fits with dtSearch.

I think the dtSearch tool that does the parsing,indexing and search based on combinations of keywords is much like the approach I was describing in earlier posts.

The product sounds like it does most everything you need. It reminds me of a product called SSN Names (or like that)?? from Software AG back in the '80s.

I looked at your Form for your User Manual and found a Missing or Misspelled form called Filter on the middle button option.

[Fipper_SG]

Yes for the missing form, I am still working on the User Manual... It's just that now that the product works, I kinda laid back and can't find the motivation to do the user manual... ahah :P

For DT Search you are right. It does make everything that you mentionned. However, you can only find a document based on the search, so basically based on a subject.

With access, if you add the main subjects to the keyword column, what we can do is filter the files by subject, but also by date, classification, type and even originator (since we put this info into the nomenclature) and this is where it becomes more useful than DT Search as we can narrow our search for a file or an information.

We do use both methods. If we just want to make a quick search we directly go through DT Search.

[orange]

Yes, there's an old phrase with computer projects that's probably as true today as it was in the 70's --the job's not over till the paper work is done. In my experience I found that the better the documentation, and better tested, the easier the maintenance, the easier to make modifications when required, and the easier to train users.

It's just that now that the product works, I kinda laid back and can't find the motivation to do the user manual... ahah :P

This is not uncommon. It just takes a bit of dedicated effort - and I would recommend you involve your team/users to identify the steps involved, get the wording so people all agree. Make it a team effort, not you alone.

I would even start with a high level flow diagram that shows the major pieces and what comes in, the blackbox (process) and what comes out. Then break that into more more detailed pictures as needed. (This will assist communications with users and management - and it will be key for maintenance/revisions/updates).

A couple of things that still jump out at me:

If you have a working solution, but feel it should/could be better, then make a list of the issues you and the team(s) see.


dtSearch -- have you talked with the techies at dtSearch (or the Canadian distributor RCC-CGA Consulting http://www.rcc-cga.com/) and told them about your process and what can dtSearch do for you? I think, from my little reading, that dtSearch can look for multiple keywords and synonyms in a single run. I also think that dtSearch can save its indexes for reuse. Only re-index if you have more documents- and even then I think you could do incremental indexing - but I don't know that for sure. (And I must say that I don't follow your processes in detail. I'm from a database background, and indexing is key to rapid search. I have also worked with some full text search tools, but they are changing so fast.)

Are you testing/trial the dtSearch or did you buy? Does dtSearch offer any training?

Access: I looked at your user manual and noted your comment about line 18. I recommend you get the free utility called MZTools for VBA (http://www.mztools.com/v3/download.aspx). It can help you with Access documentation, and assign line numbers to your procedures. You could even make the routine a Function, that asks for the filename directly, or through a form -- this would prevent possible typos.

Anyway, some thoughts to consider. And I think you could get more info from dtSearch to assist with technical questions.

Good luck.

additional info:

I found this that might help your documentation (pic) (from dtSearch)
http://www.google.ca/url?sa=t&rct=j&...,d.dmQ&cad=rja

and some videos on youtube one is http://www.youtube.com/watch?v=UyvZp7tHAqk

[Fipper_SG]

Whoa thanks a lot for the info, I'll work on this and come back if I have any question.

For DT Search, we have the full version. However, I don't know of any training. DT Search do save its index for further use. You can even save your search queries to a text file. It is a very good product for what it is designed to. For my log, it takes me less than 2 minutes to add the keywords by using DT Search (it could be a little bit longer if I had more but now I'm doing less than 20 searches). I like the way it works. It saves me a lot of time.

Right now, I'm trying to add something to the project. This has nothing to do with this subject, but I plan on putting it in the same Access File (to make a complete ''toolbox''). I'm trying to make an association matrix. In a table, I log data as:
Person A - Is the Father of - Person B
Person B - Is a member of - Organisation A

And then, a filter command (or crosstab query) could show all links about someone/something that is logged.

Have you ever heard of anything that I could use to export this to powerpoint, or anything else (persons and organisations being shapes and links being connectors)? I might be totally wrong to use access for this, but I find it very esay to use to log the data, now I'd like to illustrate it as this:
http://www.globalsecurity.org/milita...-07-22/f-2.gif

or, even better, this:
http://www.fmsasg.com/SocialNetworkA...ysis_Graph.gif

[orange]

I think your data as you have described as "associative" is very much suited to a relational database. Access or any other RDBMS can handle your "associative" data. The key to database is to design your tables and relationships to support your "business" facts and requirements. This takes skill and effort and should be done before getting into the forms and colors and what queries etc. Too many people think the software will do this for them-- big mistake. I'm on several forums, and most issues from posters relate to poor design; no real understanding of the business the database is for; lack of design principles and concepts. Sure there are some technical and philosophical issues (surrogate keys, calculated values, syntax issues, class module usage..) but these are usually people with design skills.

Your first gif is one means of presentation. There are many graphic approaches to help communicate a message - bar graphs; speedometer, pie charts, spreadsheets,....


Your second gif reminds me of some research materials we used during the conceptualization, design and creation of Strategis (http://www.ic.gc.ca/Intro.html). There were many ideas about searches; discovering nuggets of info in the masses of data people had access to. The big challenges for many organizations were finding the information you knew you had somewhere, and finding the information you didn't know you had, and relating it to various subjects in a timely fashion in a manner to convey the underlying information. One of the articles I saw after Strategis had been launched and operational for a few years described this book and the concepts-(Linked: The New Science of Networks byHungarian physicist Albert-László Barabásiin 2002) which put some of these "related topics/issues" into context. The book is by a physicist and quite deep. This sort of gif was trying to use graph theory and related mathematics to put the hidden methods/facts/terms together to show the importance(or other measure) of some topical/subjects/terminology and position that info with other subjects/terminology.

Sometimes a graphical representation can provide much more information than any other method. We had a routine(animated) that could take various sets of numbers and create "poles" and Bars and circles on a map of Canada-- that were effective in showing/analyzing the distribution of small businesses, by Industry code, by Province/City. Another I remember was a map of the USA on which the size/colour of the state changed by amount of beer sales by month (strong correlation with football and basketball); or the amount of perfume sold by retail stores by city by week and to see the correlation with Valentine's day, Christmas, Mother's day. Another was a series of network outages and duration, by day by city, that revealed major network deficiencies (contention/capacity issues; alternate paths..)

My gut feeling is that dtSearch can do more of your overall work, and it could be linked with SQLServer or MySQL if more horsepower is needed, and you could still use Access as your Front end interface. In fact the ACE/JET (often called Access) database may still be sufficient for your database needs. I would recommend you find out if dtSearch or RCC-CGA Consulting http://www.rcc-cga.com/ offer some training or technical support. You can certainly take advantage of one time major indexing and subsequent incremental indexing. I also think you could use or make more use of a dtSearch thesaurus facility if one exists. You should also see if there is a technical user group (other user/other sites) who use dtSearch -- this can be a great source of ideas and people with common interests.

Since you bought the product, I would think dtSearch or RCC would have some interest in knowing you are using the product well. Did they size up your application to their dtSearch software and make any suggestions? or did you just tryit and buyit? It may be worth some discussion with them and/or bringing in a specialist who can help you with design and usage of the product/tool and your needs.

I realize you are an Access /vba novice, but I highly recommend you do some investigation.

I know getting this system to work and improve performance is a good feeling, but I don't want to see you accept this

It's just that now that the product works, I kinda laid back and can't find the motivation to do the user manual... ahah :P

The MZTools for VBA is a good product and it's free.

Good luck.