Data People Community Competition Day 1 - Generative AI & LLMs: Opportunities, Benefits, and Strengths 💪🔝

Currently I'm working on a hobby project that uses ChatGPT to categorize astronomy news items, so that I can more easily find certain items when I'm preparing a presentation on that topic. What I've learned quite quickly is that the output ChatGPT (and probably other LLMs as well) produces, can be of quite inconsistent format. For example, when I asked it to give me 1-6 terms to tag a news item with, it sometimes delivered comma separated results, sometimes new line separated, sometimes lines started with dashes, sometimes the result was all caps.

So you really need to be very clear how you want your result. It is possible to do so. And even then there might be variations. It's almost like I outsourced my request to a team of humans, each with their own way of working.

When you start using LLM's in data pipelines, you really need to be strict in the prompt you write, decide on the “temperature” of the model (how much variation you want) and you really need to check the data quality afterwards.

Hi, I have the same experience that the output can be sometimes inconsistent. What helped me was to provide a few examples (containing the correct output format) as part of the prompt + retry the query if it failed to be parsed.

I like how you look at LLMs like on humans when thinking about that to expect. This is something unusual when we are talking about algorithms or ML but I think it can be beneficial in a lot of situations when dealing with LLMs.