Richard Goldman: [00:00:05] Thank you for joining. My name is Richard Goldman. I oversee the strategy with our hedge fund clients at LSEG. We have around 800 clients globally, and we work very closely with them. And we thought this event would be timely, because if you think about it, large language models have been around for about six years when BERT came out, and it's only been not even two years since ChatGPT came out. And obviously there's tremendous interest and the general purpose models have been integrated in so many things rapidly. But when it comes to the domain specific finance and how it's being used in finance, we're still very, very early in the game. We'll talk about some of the ways that we sort of confirm this, beyond the anecdotal that we have with conversations with clients and prospects. But we thought since we are early in the game, there's a lot to be learned by many people who we work with. And so, the purpose of this event is to try to add different perspectives to the conversations, which there are other conferences on large language models. We wanted to show a slightly different perspective, and so we brought panellists with different views to the table. And our first panel is going to be led by my colleague Nicole. So, I'll pass it on to her.
Nicole Allen: [00:01:23] All right. Thank you everybody for coming tonight. I'm really excited to be the moderator of this panel. We have great weather, wonderful venue. So, I'm a product manager. I'm head of text analytics here at LSEG. My name is Nicole Allen. I've worked in fintech for almost two decades, and I have a lot of experience working with hedge funds. I've always found working with hedge funds to be very exciting because of the nature of the challenges that you face with large data, trying to find signal from the noise. Opportunity. And I'm really excited to welcome our panellists here this evening. So first I have Dr. Sonu Chopra-Khullar. Sonu is Head of Machine Learning at Capital One. She's based in Philadelphia. Sonu has an interest in representation learning, foundational models, and she's the head of a machine learning team. I met Sonu many years ago when she was an engineer at Bloomberg, and prior to that, she was a quant trader at a Stat Arb firm, and she holds a PhD from Penn. Next, we have Dr. Aziz Lookman. Dr. Lookman is founder of rational AI and essentially rational AI was able to build into a multi-million dollar risk analytics firm. Currently, he consults for PE firms, leads data science programs, and previously he worked at Two Sigma Amazon Lending, and he has a PhD from Carnegie Mellon. And last but not least, we have Ted Merz. Ted is a founder of Principal Media. It's a writing agency for executive leadership, thought leadership. And Ted previously was global head of news product at Bloomberg, where he had a long and storied career. And he was also Managing Editor of Bloomberg News for the Americas and New York bureau chief. So welcome to our distinguished panellists, and I'm really looking forward to this discussion. All right. Interestingly, we have journalists, and we have CFA, and we have data scientists. So, you know the full gamut here. So first I wanted to just talk about a survey that LSEG did. This was in November of 2023. Essentially, we commissioned this survey with Waters Technology. And we interviewed 55 financial firms. We tried to account for bias. We didn't really look for any location or specific size. And we surveyed them. One of the main questions we asked and we're going to share this survey with you, is what do you think about the state of LLMS and adoption? And in November of 2023 it was about 50:50. We had slightly more that were using them than less, but it was still a bit of a toss-up. So, I wanted to ask our panellists, first off, are people more comfortable now? Or is there still a lot of uncertainty and if so, why? Fear? What are what are you kind of seeing in the market? So, I'll start with you, Sonu.
Sonu Chopra-Khullar: [00:04:28] I think people are finding that there's a strong need for quality checks and factual accuracy and that it seems like, oh, you can easily call an LLM. But using them to get reliable, consistent results is, I think, where the IP and the talent is.
Nicole Allen: [00:04:51] Aziz
Aziz Lookman: [00:04:52] So the way I think about LLM adoption is that it's almost that there are different types of users. So even if I take the hedge fund community or any investment community, and I split them into, for lack of a better word or classification, technically sophisticated and qualitatively sophisticated. The technically sophisticated players have been at the forefront and are increasingly comfortable using LLMs. And that's partly because they come from a tradition where they are very comfortable with signals that are noisy and they are used to the idea, okay, we're going to combine a bunch of noisy signals to get to get alpha. So, in that sort of a framework, an LLM hallucinates. Big deal. I mean, every feature hallucinates. It's no different from any other noisy feature they use. And I would say they're using it especially for feature engineering quite a bit.
Ted Merz: [00:05:50] I would just say I agree with Aziz. I put on one caveat. I think the big surprise is that there's been a decrease in expectations for LLMs in general. In my business, a number of my customers will say they will ask for guarantees that I'm not using ChatGPT, I'm not using it. And so, I see that's a real surprise, given the way we all think it's going to be very revolutionary that there's I think there's been a decline in expectations and a bit of concern about the output.
Aziz Lookman: [00:06:22] I mean, everyone's seen this sort of the hype cycle, the irrational exuberance, the trough of disappointment. So, the way I almost think about it is that the two camps I mentioned, both went through it, but the amplitude was a lot less for the technically sophisticated ones. Why? Because, I don't know, maybe we are quants, we’re used to being disappointed. So, you know, you say, yeah, it's not going to be that great, but we're going to get that disappointed either.
Nicole Allen: [00:06:50] That leads to my next topic. So, LSEG also worked on a white paper with the head quant of probability and partners. Her name is Dr. Svetlana Borovkov. And essentially, we're going to provide the link to this white paper, but it's a very good primer of LLMs in specific to finance and some of the challenges and opportunities. So, when she wrote this paper, she talked about uninformed enthusiasm and a lack of deep understanding. But then she also said a year in that we're taking a more measured approach. So, what has been your personal experience as it relates to working with some of these newer types of generative AI, and maybe what you've encountered in your daily life and at work?
Sonu Chopra-Khullar: [00:07:37] I think that it does take some experience to use these models well, and you don't have to be terribly brilliant to use them well, but you do have to have a bit of experience. And I was quite surprised by that.
Aziz Lookman: [00:07:52] So it's been pretty interesting. I've been doing this now somewhere for between 15 and 20 years. And, you know, you try to have a conversation with a CEO, and you try to get him excited about linear regression. And it's about as successful as getting a kid excited about broccoli. Now, you know, the best thing to my mind about LLMs, it's like it's a gateway drug. You the CEO see it. They get super excited about it. They say, I want to do this. I want my memos to be generated automatically. And then that's, you know, you got to be like the smart dealer here. You got to say, okay, that's cool. And in order to do it, I need, you know, I'll give you a memo. But in order to get the memo, I need, you know, I need features, I need this, I need that, and then you can build a linear regression model and get something done. So, I think there's two things right. One is as a practical matter I think it's really changed the level of enthusiasm I see for quantitative methods generally. And to me that's a really good thing. People who would summarily discount any quantitative method as saying, you know, this is basically BS. I've done this for 20 years. What are you going to teach me? All of a sudden, they interested in using a machine to help them make decisions. I think also on a practical level I'm seeing use really across the front office, middle office, back office in investment services firms. And that's both I mean, I mentioned how quants are using it for feature engineering, but even compliance folks are using it for stuff like keeping track of regulations changing. And this is real. People are using it today.
Ted Merz: [00:09:28] I would just say, and I think we'll get to some examples later, but I would say that the most interesting thing is that it has eroded the moat around every business. And that's what strikes me, is that there's an ability to use these in a way that you can build new businesses. And so, the incumbents are incredibly vulnerable, and that has not yet really come to the fore. But that's what I see happening.
Nicole Allen: [00:09:52] So, what about the practical applications? Ted, I know you have some examples.
Ted Merz: [00:09:56] I have a slide.
Nicole Allen: [00:09:57] Yeah. Okay.
Ted Merz: [00:10:00] So, the practical thing, the most incredible thing about large language models, and particularly ChatGPT, is two years in how little it's being used. I asked Grok, which is my favourite LLM, you know who is the host of the All in pod? The all in pod. I hope some of you guys know. Most of you know, probably the biggest business publication. So, you can see the names. If you can't, it's Chamath [Palihapitiya], Jason Calacanis, David Sacks and Bill Ackman. So now if you're going to get that wrong, that is the reason. It's the asymmetric risk for companies to get that kind of thing wrong. And that's really where we are right now. They don't really know how to deal with that. So, I guess I wanted to use just a couple minutes to say where I'm seeing all the interesting or a lot of interesting growth is at the Start-Up level, and how they're using it in different ways. I'm just going to give three quick examples. One is the most common example, I think, for Wall Street was initially, how do we summarise large amounts of information and distil it into bullet points. And so, lots of people are working on that. One of the companies I've had some interaction with is called Dotata, and it's basically what they do is they take transcripts, conference calls, they run them through an AI model, and they produce a summary. What's interesting is that they're now at the point where you can ask it something like the question I asked this morning was, where is Chevron focusing its engineering or its exploration efforts, and how has that changed over the last ten years? And boom, it tells you they're focusing in Africa and Ghana and the Permian Basin in these places. And that's the kind of question that would take days for a hedge fund analyst in the past. The key point I would say about why it's working is because they have, it is not open ChatGPT. They've preloaded a cohort of the conference calls into a set, and they've trained the model with a domain expertise to do that. So, the hallucinations are very low. So, I think probably the early gains particularly hedge funds and a lot of the people in the street are going to come in that kind of summarisation. The second example is a company called Pricing Culture. Pricing Culture started out by gathering data sets. They were collecting data sets of collectables, and then they realised that those data sets are very arcane and hard to access, and nobody understands what they mean. So, they used AI to generate text stories to make that data accessible. Probably the biggest, biggest implication for people here are lots of us here use data sets. They're often arcane. Very few people can understand what that data means. So, if you have a very obscure data set, the way it becomes accessible to a larger number of analysts is to essentially turn it into text. And it allows you to put in context. And the third example I'd put, and I think these are kind of the three big examples, is a company I've been looking at called SigTech. SigTech is basically trying to create an AI version of quants, and so it's funded by Alan Howard from Brevan Howard. And the idea of the firm is that basically you use AI again with limited data cohorts to be able to generate what previously would have been required a quant to create. And I think that's like an incredibly interesting arena. Those three applications are, for me, what I've been looking at. I posted all these on my Twitter account today with examples. So, if you go to my Twitter, you can sort of see them if you want to look at those. But that's kind of what, what I'm seeing those three big use cases.
Nicole Allen: [00:14:02] That's awesome. Okay. Aziz, maybe you can talk a little bit about as well. And Sonu perhaps some intriguing or more innovative applications that you've found to be really unique and or any tools or things that have really impressed you that you think would be useful to the audience here today?
Sonu Chopra-Khullar: [00:14:21] Insomuch as I'm allowed to talk about anything proprietary, which I'm not, I'm going to be speaking not as a Capital One employee, but as an ex quant. People talk about text and natural language and summarisation, but I think in the hedge fund space, if I were trading, which I'm not, but if I were I'd start looking at different ways of words and documents and I would look at transactions, which I'm sure the hedge fund folks are doing. And I would look at embeddings of transactions in documents of transactions, and I would look at that to find signal. And I'm sure lots of people already are doing that, but it's taking it away from natural language and putting it in the context of, you know, trades. And again, that's not Capital One, that's just Sonu.
Aziz Lookman: [00:15:10] So, same caveat holds for me. I'm not speaking as an ex Two Sigma employee or as a consultant to the companies I am currently consulting. I'm simply talking as a whatever a student in this area. The applications that I find are interesting aren't ones where it's, it's cool, but where it's well engineered. The way I think about it is the applications that seem to be pretty good are ones where you could have got to the same outcome, say, using an army of second year summer interns, but it would be phenomenally expensive to pay those interns. So typically, if you can set up your use case, so the level of sophistication required per unit of activity is low. But you get a sophisticated product by being lots and lots of units of this activity. That's where I've seen success. So, the kinds of use cases I've seen that I thought were pretty cool were, for example, say, summarising what a million companies do. Now, why is that useful? Because once you've got, say, information on a million companies, then you can look at things such as market concentration. What are the factors driving corporate dynamics? What are the factors driving the macro factors driving the industry demand? Things like that I’ve seen equally good use cases on things such as compliance. So, it sounds pretty unsexy. But if you think about a multi-strat fund for the LPs, what they make is investment alpha minus operations costs. So, if you bring down your operating costs, you're basically creating alpha for your LPs. Now how can you bring that down? One example in compliance is you often have to respond to RFPs, how is your fund whatever compliant with some ESG mandate. That stuff typically is answered by a lawyer being paid several hundred bucks an hour. I know that BNY Mellon is on record saying that they have got multiple use cases. I don't know if BNY Mellon has done this, but I know of others that have where they've basically got the LLM to provide a first draft. So now you're saving money, which means you're going to send money back to your LPs in the form of a couple of basis points. And your LPs don't care whether you came up with a smarter trading idea, or you came up with a way to save them operating expenses, you're putting more money in their pockets.
Ted Merz: [00:17:33] Like I love that, Aziz. I love the idea that you're looking for examples where you spend a lot of time and money to do something, and if you can do it inexpensively, I think that's great. So, there's one other company I'm going to mention that I'm aware of. I've talked to the founder. His name is Sean Austin, and he runs this thing called Markets EQ. And the clever idea is they record all the conference calls and earnings calls and all these things. And they run it through this large language model that detects, it analyses the tone in the voice. So typically, people look for sentiment in things. But even on a more basic level, he just counts and produces an analytic on the speed at which the CEO speaks. And so, one of the things you realise if you've worked in a big company is that PR IR departments spend weeks preparing for the earnings call. They're coaching them, they're practising, and this is going to become like an analytic where you when you do the dry run, they measure you and they're like okay you're talking too fast. That's a sign you're nervous. And you're going to be scored. I think the way kind of like it's going to change the way people communicate, the way we've used AWS on football. I don't know how many people are football fans, but like all the statistics that have come out of that and the ability to measure. I think we're going to see these crazy levels of statistics, which in the past you would have just listened to the person and said, you're talking too fast, you're whatever, but now you're going to know, based on 20 years of calls you've done, the rate at which you're talking and like how it's changed and how your peers talk and everything else. So, things like that, I think are really exciting. And they're all happening at these like niche use cases.
Nicole Allen: [00:19:11] It's fascinating. Okay. So, any good examples perhaps using a combination of like structured and unstructured data that you might want to mention? I'm somebody who works in news products. So, I'm always interested in unstructured data combining with numeric indicators, especially given that, we're talking about GenAI, which is all focussed on language and kind of combining that with more traditional AI models.
Aziz Lookman: [00:19:39] This goes back to how I think about it. If you had an army of second year interns, what would you do? You'd say, look, I want to create another feature for my model. Come up with a feature that, let's say, mentions the number of times company A is mentioned next to company B. Now that sort of a use case I think has become increasingly common. In effect, it's using the language comprehension ability of LLMs less the generation part and the and the basic pipeline is take unstructured data, put it through an LLM, make it into a structured feature and add that structured feature to all your other structured features in whatever other modelling framework you have. So that paradigm I've seen time and again.
Nicole Allen: [00:20:26] Any specific tools, perhaps that you like that somebody in the audience might be, might find useful that you've come across recently. I know we talked about different examples of LLMs and use cases, but.
Ted Merz: [00:20:44] I don't know. I mean, there's like it's I think we're at this point. I mean, I like Google Docs. Anybody use Google Docs? But what's interesting is all your major platform things you're supplementing now with a million other things. So, like I use for obviously when you do a video, I use Otter to get the transcript, but there's dozens of other applications that do the same thing with nuances. I don't know, I think we're in a weird space where just like we have all these LLMs and you, you use Claude, and you use Perplexity and you're looking at differences and you can't explain it. You use Grok, and you're like, you can't explain why they have differences. I think we're at that point in the cycle where there's a million car companies and there's only going to be three sometime from now, but I'm not sure who's going to win? I don't know. I just think it's very, very interesting. Lots and lots of Start-Ups are producing these. They, for the most part, have not been integrated into your Microsoft platforms or your Google platforms or any other major platforms. All the useful things.
Nicole Allen: [00:21:49] Okay. That's great. I mean, that kind of leads to speed of adoption. And I mean, none of us have a crystal ball. But in terms of what you're seeing now and what you expect a year from now, what do you think is going to happen? I mean, do you do you feel like we're at the point where people kind of know what to do with existing technology, and we're waiting for the next best thing, or are we waiting for costs to come down? Should we be planning? What do you think we should be doing right now to prepare as technology is changing. Aziz.
Aziz Lookman: [00:22:21] I mean, I think this is a well-known statistic. I think the cost of LLMs have come down something like about 80% in the last one year, and all indications are that trend is only going to continue. So, I think what we're going to see is if you can imagine a use case where you can't do it today because it's too expensive, because you need to make too many LLM calls, wait for a couple of months, and that's probably going to become cost effective. So, I think if I had to think about building a product, I'd almost build, assuming it's not financially viable today, but will be financially viable if the cost per token goes down by 50%. I think that's how I'd plan. If I may go back to your, you know, tools. I tend to interact with LLMs at a programmatic level. Just I mean, what can I say, I'm a nerd. So I probably mention two things. It's pretty cool nowadays. The level of, I guess, AI assistants you can get when you're coding. I mean, at least I don't remember Python syntax. So, I've got to, you know, normally you'd go to StackOverflow or whatever. It's not like the LLM is doing something brilliant, but it's saving you time and it's really reducing the frustrations for somebody who's trying to get up to speed on coding. So if you know, you have people on your team saying, oh, I want to get into coding, I would really encourage them to take an LLM assisted route. I know that a deeplearning ai has something that they're offering along these lines. The other thing I personally like is to use some sort of a platform that's LLM platform agnostic. So, any of them, like you use Lang chain, so on and so forth. What's nice about them is you can do all your work and then say, oh, now I don't want to use OpenAI's product, I want to use Google's product or somebody else. It's really quick to swap in and swap out. And that's useful because sometimes you'll find that for your use case, OpenAI is giving you 90% accuracy. Gemini is giving you 99% accuracy. Why? Who knows? But in some sense, who cares, right? So, I think those are the two things I would do.
Sonu Chopra-Khullar: [00:24:36] I would actually have a counterargument. I think tag sets are specific to models. Mark-up and tag sets are specific to models. And so, I think they're not quite as interchangeable as one might be led to believe.
Nicole Allen: [00:24:50] If I were a hedge fund, what should I do now? This is a very open ended question. What advice would you give? I mean, I feel like we've already kind of given some commentary around what we think is happening. Experience. What would be like your lessons learned or advice that you would give. I mean, there's lots of roles at a hedge fund as well of course.
Sonu Chopra-Khullar: [00:25:13] I remember when people talked about song embeddings at Spotify. Right. And the words were the notes of music, and the documents were the songs. And I sort of said this before. I think thinking in different contexts beyond natural language is where the edge is. Again, speaking as a private consumer and not anything I've ever done at Capital One, I think that's where there might be trading signal.
Ted Merz: [00:25:40] Can you say more about that? I don't. What do you mean by that?
Sonu Chopra-Khullar: [00:25:43] So I mean, you know, when you talk about trades in a sea of trades, like what do high frequency traders do? They look for signal, right? So that's your embedding. Where is your signal in that. You know, documents of trades. Similarly. when people try to analyse supply chains and, you know, jump the supply chain to see like, oh, can I trade on this unknown signal earlier in the supply chain. I think just taking it and putting it in a different context, you know, as people have used LLMs for protein synthesis. Right. It's not just words. It can be other things. And I suspect that that's where the edge will be.
Aziz Lookman: [00:26:23] Oh I see. So, use the same architecture but to process something other than words.
Sonu Chopra-Khullar: Yeah, yeah.
Ted Merz: [00:26:31] A lot of thoughts. But the first big one is that I used to work at this company, and the moat around that company was really the fact that they had collected data for 40 years, and it was structured, and each data was in a data field. And 3 years ago, you would have said it is impossible to replicate what they've collected over 40 years, putting it in the data field and scrubbing it. So, the first thing about ChatGPT or not, ChatGPT, AI and large language models. The first is that it opens up the world beyond nonfarm payrolls to a bazillion other non-structured alternative data sets. And I think the first thing is we're thinking about alternative data way too narrowly, and it's going to be much, much, much bigger. And we're able to use these tools and agents to basically not only replicate all the existing data that was in the big platforms but create lots and lots of new data sets. And so, in a way, and I was talking to a Start-Up yesterday that was basically thinking about how to build agents that go out all the time and just look for data. And when you think about like, what Rentech did and all the effort they took to go get specialised data sets and put them together and run them. Imagine if you're really your job is not to go and get a specific data set, but to think about how to set agents upon the world to find data and bring it back. And you don't even know what data you're looking for. So, you know, Rentech was like, let's get Peruvian cement data and look for correlations. This is different. This is like looking for data, you don't know what it is, but so you have to think about the world and a little bit differently. And I think that's going to be a huge opportunity. Most of those data sets are also going to become available in weird ways, where it's not like you go and you have an API to buy them. That's kind of where we are now, but they're going to be potentially available where again, you set an agent out to go find it and suck it in, and it's going to be very, very interesting. The agent itself is going to try to figure out if it's useful or not. So, I think you have to really think about how you're getting data, particularly alternative data differently for your models and other things.
Aziz Lookman: [00:28:42] So specifically what I would do is I think I would give every employee like a $5,000 budget with basically no strings attached. Do something cool with LLMs. And I won't say if I know or do not know of one that’s done this.
Nicole Allen: [00:28:58] I hope you have a budget.
Aziz Lookman: [00:29:01] Okay. I mean, whatever it is, if it's not five grand, one grand, but actually give people the ability to experiment and come up with interesting use cases. So that's what I would do. The other thing I would do is figure out governance. Again, I really think the world seems at least the few examples I've seen, it seems to be the technically sophisticated and the qualitatively sophisticated. The technical camp says, look, you're an adult, you're going to do sensible things. We've already put you through all the compliance stuff. We do it yearly. Don't do stupid stuff. And the other camp seems to be, this is oh, this is like some, you know, dangerous technology. We don't trust you to do it. And we're either not going to give you access or we're going to hobble it to such an extent that you either won't use it or it's not fun or not useful. So, I think you need both. You basically need to empower your employees to use it. If I may just go back to a point Sonu made about using the architecture, I think that's a really cool one. I, in fact, know of someone who's using transformer architecture for pricing products. So, context is you have several thousand products, so say you're trying to sell a 32 ounce jug of liquid soap and a 16 ounce jug of liquid soap. They're kind of related, but how do you think about relationships that transform architecture provides a nice, at least a nice starting point to think about relationships and therefore relationships and how they are priced. So, I really, I really think that's a neat a neat point. Thanks.
Sonu Chopra-Khullar: [00:30:34] And you don't even have to use transformers if everyone remembers Word2vec from 2013. The idea was just simply, you know, a word's meaning is derived from its context. You don't need attention. You don't need, you know, multiple head attention. I think even just the simple architecture from a decade ago, over a decade ago is relevant.
Aziz Lookman: [00:30:56] But Sonu, you wouldn't be a cool kid anymore.
Sonu Chopra-Khullar: [00:30:59] Okay. Sorry. Mostly uncool. Yeah. As my kids would already say. Yes.
Aziz Lookman: [00:31:02] Same here!
Nicole Allen: [00:31:03] What are the biggest challenges you’ve faced and how have you overcome that? You know, in terms of the data that you've worked with and knowing where we are with the technology right now, any interesting solutions you've been able to figure out to help you?
Sonu Chopra-Khullar: [00:31:20] I'll just say that, again, without commenting on specific things that I'm working on. I think things can seem deceptively simple. I mean, I know I was just humbled by things that seem deceptively simple, and there's so many ways that it's, like, deeply fascinating. Things that seem conceptually simple. There's just all these rabbit holes and warrens and interesting things. And I found to be I just am so engrossed by everything that I learn every day. And I guess I'm fortunate that I get to see that. And I was humbled by it, actually.
Aziz Lookman: [00:31:54] I think I would second that. It's easy to come up with what looks like really cool use cases. But I think the in some sense the way an LLM breaks down. There are more ways in which an LLM can break down than say other models can break down. So, once you start deviating from your nice neat, cool demo case, it's a lot easier to find less than optimal output. So, what that means is it's really like going back to modelling 101. You have to think carefully about what your model is. What's it going to be used at, how do you build sensible guardrails? How do you think about edge cases? How do you actually train this model to do what you want it to do? I think the more we disabuse ourselves of this notion that, oh, this is, it's almost like saying, look, I want to build a linear regression model of data X, to predict Y using some X. And I don't actually need to train my linear regression model. I'm just going to get an API for linear regression. I'm going to get a sensible answer. It's kind of nuts to expect it, but that's the expectation with LLMS.
Ted Merz: [00:33:03] I'm going to take it slightly different. So, my main business is writing on behalf of people. And you know, we're in this world now where ChatGPT can produce an endless amount of content. A lot of people, when they write with ChatGPT, look at it and say, that's awesome. And you know, why did I need to spend any time writing? Because it's really good. What the data shows, and this is like one of the things you see on LinkedIn, the engagement rates on ChatGPT written content are like very low versus human written content. So even if you look at a piece of content and you think, wow, no one's going to know, you know, they know, like even if they look it and they say, I can't tell they know in somewhere because they don't click on it. So, one thing I would say to, I guess you asked about like just generally hedge funds, but this is everybody. Don't overestimate what the LLMS can do. And this isn't just about writing, but it's about everything. I think there's an element of being efficient and an element of being effective. So, if you are efficient in that ChatGPT or something like it can do 10,000 emails for sales leads and you can only write ten. It's possible that the ten are more valuable to you than the 10,000, even though it seems less efficient, because that's about being effective, not being efficient, if you know what I'm saying. Like when you're trying to actually raise money or something, you're trying to land a client. Sometimes the most efficient is not the most effective. And I think that I would say that in general, that's a that's a thing to be aware of and the limitations that people don't know because they haven't used them very much.
Aziz Lookman: [00:34:43] To reiterate what Sonu and Ted said, I think there's a little bit of thinking that the LLM is the ultimate tool. It can do everything and it's a little bit nuts. It's like saying, I want I insist on building a spreadsheet model using Microsoft Word. I mean, why would you do it? I mean, you'd probably use Excel instead. So, I think it's also thinking about using the right tool for the right job. LLMs aren't for everything. So, we keep seeing people saying, oh, let's use an LLM to solve a math problem. Why?
Nicole Allen: [00:35:15] It goes back to the point about uninformed enthusiasm.
Aziz Lookman: Yeah
Nicole Allen: Yeah. All right. I got the time signal over there. So, we're going to we have we have a little bit of time for questions. I'd love to open questions in the audience.
Audience Question (quiet): [00:35:30] On the point of ?? generating an endless amount of content. There were multiple examples of like going out and finding data and looking for signals in the data. What are your thoughts on like this cannibalism of the model consuming model generated?
Ted Merz: [00:35:47] Yeah. Yeah. Like the cow eating cows. Yeah. The question is, what happened? Oh, this is a really good question, which is that we don't have enough language to train the language models. So, one expectation is we're going to generate more language with ChatGPT. And that will be used to train the models. So that is kind of the cows eating cows. And the question is what's the implication of that. So, I will say but I'm not technical, so, I want you guys to say something about that hopefully. But I think what seems obvious when you read a lot of this is it drives it to the middle. You know, it drives it to the average of some sort. It's just what ChatGPT does. Like when you ask it a question off the off the shelf model, is it just it blends all those and puts it into this kind of a neutral middle sounding tone and maybe prompts can overcome that, but I think generally not. I mean, generally that's what I think is going to happen. And as you feed more and more language into that, that will become the case. And so, what will happen is the outliers, the long tail stuff is what, depending on your use case, will be helpful to you. So, it depends what you're really using it for I think. But my point is like so if I'm writing a letter to Sequoia to get funded, I don't want it to sound like everybody else's letter, you know? So, I'm going to be a long tail on that. And I think that's the risk if you use it, and it sounds like everyone else is going to be a problem. But mathematically, I'd like to know what you guys think.
Aziz Lookman: [00:37:14] So I have a slightly different take. If you're trying to build a model that's going to answer every question for everyone. I'm not smart enough to do that, so I can't help you there. But what I've noticed is increasingly the this the field is moving to have more niche applications and in the niches there's still lots of unused data. So, we probably are running out of data that's in the private domain, I'm sorry. In the public domain, there's still a lot of data in the private domain, and it's typically enough to train the model for the use case you have in mind. The other thing that we are increasingly starting to see now is, in a sense, training a model with a human in the loop. So, it's, to my mind, it's similar to the old idea we always had of being Bayesian. You start off with a with a model by having some priors. This is similar in that spirit saying, no, I think this is how it should work. And I'm going to basically impose my beliefs through feedback to get the model to act the way I want it to work.
Sonu Chopra-Khullar: [00:38:16] I'll just add that outside of the context of LLMs and again, not related to anything I do at C-One. You know, synthetic data does is useful. People train classifiers outside of LLMS, and when you provide synthetic examples, you can get a more powerful classifier. I think the previous question was, would human knowledge sort of trend towards mediocrity if everything was machine generated? And we're training on machine generated content? Maybe. But then synthetic data has had success.
Nicole Allen: [00:38:48] Okay. I think we've reached time. So, we're going to move on to our next panel, but I want to thank our distinguished speakers once again for joining this panel, and I hope everybody found it interesting.
Disclaimer [00:39:08] The information contained in this podcast does not constitute a recommendation from any LSEG entity to the listener. The views expressed in this podcast are not necessarily those of LSEG and LSEG is not providing any investment, financial, economic, legal, accounting or tax advice or recommendations in this podcast. Neither LSEG nor any of its affiliates makes any representation or warranty as to the accuracy or completeness of the statements or any information contained in this podcast and any and all liability therefore, whether direct or indirect is expressly disclaimed. For further information, visit the show notes of this podcast or lseg.com.
