CSV to Excel

Project I subscribe to a data service (IQFeed) that provides live intra-day data. Using a program called Qcollector, individual stock data is collected and stored locally in individual CSV files (one for each stock). These files are updated frequently (at least every 5 minutes). I need to have three databases built that maintain price data locally for 2000+ stocks on a continuous basis and would like to have a function built enabling MS Excel to read data from the databases very quickly. Data should be initially collected in three databases reflecting three different time frames – 5 minute data, hourly data, and daily data. If just one database were built, too much intra-day data would need to be stored to capture 2500 periods of end of day data. The database needs to run in the background collecting current data from the CSV files as well as historic data. It should not require intensive processing capacity as other applications will be running concurrently. The only data that needs to be collected is closing price – a single data-point for every period for every stock CSV Files Each security has its own CSV file. The location of this file, and the name of this file will not change. All CSV files will have a similar file name ([login to view URL]) and will all be stored in the same folder. A sample file is attached ([login to view URL]). Not all CSV files will have data for exactly the same periods – some might not be as current as others and not all will have the same starting point. It is likely that all will be updated every 5 minutes to reflect new data, but this is not guaranteed. Data in the database needs to be vetted to ensure that there are no missing periods. Comparability of data across securities is essential. If a data point is missing, the previous data point should be used. If a CSV file is subsequently updated, including previously missing or errant data, the database should be updated to include this corrected data. Data is time stamped at the end of the period (ie the last price between 9:30 and 9:35am is stamped 9:35am) Some CSV files will have 24 hours of data per day (currencies – 7pm Sunday through 4pm Friday) while others (stocks) will just have data between 9:30 and 4pm. In the future, foreign securities will be added that have yet different trading periods due to time zone differences. The following information will be provided in an excel spreadsheet ([login to view URL]) which should be used to determine which securities are to be maintained/updated. · Security Name · Security Type · Security Industry · Stock Sector · File Name\Location · Times for which data is collected (ie 9:30 to 4:00, 24hrs) Any additions/deletions/changes would be reflected in [login to view URL] Each CSV file is updated frequently and therefore can't be opened – it must either be queried with-out being opened, or be copied to another location, opened, and subsequently deleted Database(s) Initially three databases need to be built to hold Minute, Hourly, and Daily data – 2500 periods of data are needed in each. The data for all three databases will come from the same CSV files. Data from the hourly and daily files can come from the minute database or from the CSV files Data in the current databases can be limited to 2500 periods, however historic data should be saved in some manner. The amount of data stored in the current database should be flexible (>2500). Current vs. Future/Incremental timeframes - initially 5 minute data will be used, however this may change to 1 data in the future. Hourly and daily are likely to always remain. Databases should be updated frequently reflecting the data in the underlying CSV files (update cycle should be variable) The daily and hourly databases should use the most recent minute datapoint as the last “hour” or “day” (ie use the 2:45pm data point as the end of the day point until 3pm for the hourly database and until 4pm for the daily database). Additions/Deletions/Changes will need to be made reflecting the fact that certain stocks will be added over time and others deleted as per the [login to view URL] file The databases should assume 5 business days per week; however, a manual override should be used to allow for holidays when data would not be captured. The databases need to be transferred easily to other computers such that other computers can read the data locally. Import into Excel Build a function that pulls in at least 500 data-points from any of the databases using the following inputs (the import function should look to cells in a spreadsheet to get this information): · Stock ticker · Periodicity (5 minute, 10 minute, hour, 2 hour, day, 3 day) · Starting point (most recent period or historic) · Number of periods · Daily period (9:30 – 4pm or 24hrs) See “[login to view URL]” for a basic file showing inputs/output. Note that date/time data is optional, and that multiple time frames could be drawn from one database. Already constructed Excel macros will be used to pull in data for multiple stocks and multiple timeframes at once in rapid succession. Hence, any functions built to pull in the data from the databases must be callable by a macro. Stock name and historic data is only output required (time data is not required if it slows the function down). Data should always be sorted from most recent at the top to older at the bottom.