Running on raw log data

[mykolav95]

Is proposed solution intended to work on raw log data?

[khawar-islam]

@mykolav95 @vanhoanglepsa

How to test on raw log and how to produce output in meaningful manner not like just accuracy mentioned?

[mykolav95]

@khawar-islam Hi. Thanks for answering.
Yes, exactly.
I've had a quick grasp over the code. Seems to be that input data must be at least labeled for timestamp, severit, content, is that so?

[khawar-islam]

Dear @mykolav95

How we can generate a label data that contain timestamp, severit and content?. We have raw log data so how we can process it?

[mykolav95]

@khawar-islam I might be absolutly wrong then. Can you help me run the model on raw data?

[vanhoanglepsa]

Hi @khawar-islam , @mykolav95
LogPPT can work on both raw log data and semi-structured log data (with log headers).
Please refer to this to define the header for log data.
If your log data doesn't contain any header, just set log_format = "<Content>".

[sxj19980519]

Hi @vanhoanglepsa
Can you describe how to use it? Like what code to keep and what to get rid of

[vanhoanglepsa]

@sxj19980519 could you provide some examples of your data? basically, you will need to define header fields and some pair of (log content, log template) to run LogPPT.

[sxj19980519]

@sxj19980519 could you provide some examples of your data? basically, you will need to define header fields and some pair of (log content, log template) to run LogPPT.

Thank you for your reply,I have alread solved this problem.But when I switch model from roberta to Bert.The terminal report “RuntimeError: The size of tensor a (1576) must match the size of tensor b (512) at non-singleton dimension 1”.And I found some special long logs casued this problem.
For example, data with an index of 1580 in the dataset(HDFS_2k.log_structured)

[Tasneem91]

@sxj19980519 would you please provide the way that you solved the problem, I am trying to run it on my own raw log file but it does not work because I do not have the labeled data

[rustamtemirov]

@sxj19980519 would you please provide the way that you solved the problem, I am trying to run it on my own raw log file but it does not work because I do not have the labeled data

Could you resolve the problem?
I have raw data, but for running the fewshot sampling, it required already structured or parsed data. I don't see the purpose for using another log parsing method if LogPPT is esentially made for log parsing :?

[rustamtemirov]

@vanhoanglepsa Hi Van, could you give response on how to create the structured and structured_corrected csv files?