Quantcast
Channel: Desktop topics
Viewing all articles
Browse latest Browse all 213819

Combine CSV files without top N rows (~ IIS log files)

$
0
0

Hi, I tried to combine IIS log files into one data set, but still with no luck. The problem are comments like this at the beginning

 

#Software: Microsoft Internet Information Services 8.5
#Version: 1.0
#Date: 2017-01-18 00:00:00
#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken

 

When I click on "Combine Binaries", a dialog is opened and the engine tries to guess structure of the CSV file. This is fine. But when selecting "space", it takes the first row "#Software: Microsoft Internet Information Services 8.5" and makes a guess there are only 6 columns.

There are much more columns in the IIS files of course. But I can't workaround this feature. 

 

In ideal world I would need to

  1. skip first 3 rows
  2. - take line "#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken"
    1.    - strip #Fields"
    2.    - use the remaining items to as column names
  3.  - do it for every file

The combine itself is described here https://sharepointmike.wordpress.com/2016/03/19/loading-and-combining-multiple-csv-files-in-power-bi/ but the rest. Any help is really appreciated.. as well as pointers to doc, forums or whatever. I tried to search how to analyze log files in Power BI but with no luck.


Viewing all articles
Browse latest Browse all 213819

Trending Articles