Merging Files with similar structure but removing duplicate headers

I often get text files of similar type (Tab delimited, CSV, etc.) but they all have a header row.  A super easy way to merge the data together is to

  1. Created a folder that only has the files you want to merge
  2. create a command prompt in the directory
  3. use the following dos command to merge the files (adjusting the extension to your purpose:  copy *.csv  merged.csv

This will merge all of the files ending in .csv into the merged.csv file however it keeps all the header rows. 🙁   This annoyed me so I wrote a short program to use AutoHotkey and only keep the first header row.

Merging Files with similar structure but removing duplicate headers

Here is the code for Merging Files with similar structure but removing duplicate headers

#NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases.
#SingleInstance,Force
SetBatchLines,-1
;********************Merge files with the same Headers***********************************
;*******************************************************
;********************Get downloads folder*https://www.autohotkey.com/boards/viewtopic.php?t=66133**********************************
FOLDERID_Downloads := "{374DE290-123F-4565-9164-39C4925E467B}"
RegRead, v, HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders, % FOLDERID_Downloads
VarSetCapacity(downloads, (261 + !A_IsUnicode) << !!A_IsUnicode)
DllCall("ExpandEnvironmentStrings", Str, v, Str, downloads, UInt, 260)
FileSelectFile, Files, M3 ,%downloads% ; M3 = Multiselect existing files. set default path to downloads folder

Cancelled(Files) ;Check to see if user selected any files, if not, cancel
;********************Now loop over files, get paths, then combine***********************************
Loop, parse, files, `n
{
if A_Index = 1 ;get path
Folder_Path:=A_LoopField "\" ;Store path to folder with backslash
else { ;If the second one, read the whole file plus header
If (A_Index=2)
FileRead, Data, % *P65001 Folder_Path A_LoopField
else{
Loop, read, %Folder_Path%%A_LoopField% ;Loop over Additional files
{
If (A_Index=1) ;If header row
Continue
Data.= A_LoopReadLine "`r`n" ;Append each row
}}
}
}
InputBox,New_File,New File Name,What do you want the new file name to be?`n`nBe sure to include the extension! ;Get file name with Extension
Cancelled(New_File) ;If they don't give it a name, cancel
File_Name:=InVaild_FileName_Fixer(File_Name)
FileAppend, %Data%,%Folder_Path% %New_File%,UTF-8 ;Write file
;~ DebugWindow(data,Clear:=1,LineBreak:=1,Sleep:=500,AutoHide:=0)
Run %Folder_Path% %New_File% ;Launch it in default text editor
return

;********************See if the user took Action or hit cancel***********************************
Cancelled(Var){
if (Var =""){
MsgBox, The user pressed cancel.
ExitApp
}
}

Comments are closed.