parsing

[adrianpuiu]

Hi , can any of you guys give me an example of just parsing muliple URLs from a file without having to emmbed using ollama ?

[VinciGit00]

Use a for loop

[adrianpuiu]

neah, it wont work .the site needs to be scraped with selenium. i'll wait until it gets implemented

from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai.utils import prettify_exec_info
graph_config = {
    "llm": {
        "model": "ollama/llama3:8b-instruct-q8_0",
        "temperature": 0,
        "base_url": "http://localhost:11434",
        "format": "json",  # Ollama needs the format to be specified explicitly
        # "model_tokens": 2000, # set context length arbitrarily
    },
     "embeddings": {
        "model": "ollama/nomic-embed-text",
        # "temperature": 0,
        "base_url": "http://localhost:11434",  # set Ollama URL
    }
}


with open('cleaned-urls.txt', 'r') as f:
    sources = [line.strip() for line in f if line.strip() != '']

for source_url in sources:
    smart_scraper_graph = SmartScraperGraph(
        prompt="find the best way to extract data, eliminate unneccesary fields and organise to only show the entire conversation and code snippets. make sure to include all text from the conversation and the users answers. always the first text is the question and what follows is from other user replies",
        source=source_url,
        config=graph_config
    )
    result = smart_scraper_graph.run()
    print(result)

i get this error 👍

File "F:\maestro\WebScraper\WebScraper\LLMScrapeBS.py", line 28, in <module>
   smart_scraper_graph = SmartScraperGraph(
                         ^^^^^^^^^^^^^^^^^^
 File "F:\maestro\WebScraper\WebScraper\Scrapegraph-ai\venv\Lib\site-packages\scrapegraphai\graphs\smart_scraper_graph.py", line 47, in __init__
   super().__init__(prompt, config, source)
 File "F:\maestro\WebScraper\WebScraper\Scrapegraph-ai\venv\Lib\site-packages\scrapegraphai\graphs\abstract_graph.py", line 48, in __init__
   ) if "embeddings" not in config else self._create_embedder(
                                        ^^^^^^^^^^^^^^^^^^^^^^
 File "F:\maestro\WebScraper\WebScraper\Scrapegraph-ai\venv\Lib\site-packages\scrapegraphai\graphs\abstract_graph.py", line 288, in _create_embedder
   raise ValueError(
ValueError: Model provided by the configuration not supported

ollama list
NAME ID SIZE MODIFIED
gemma:2b b50d6c999e59 1.7 GB 44 hours ago
llama3:8b-instruct-q8_0 5a511385d20f 8.5 GB 15 minutes ago
llama3:latest a6990ed6be41 4.7 GB 2 days ago
mistral:latest 61e88e884507 4.1 GB 44 hours ago
nomic-embed-text:latest 0a109f422b47 274 MB 2 days ago
phi3:latest a2c89ceaed85 2.3 GB 45 hours ago
sroecker/granite-7b-lab:latest 206d6e1e0e42 4.1 GB 42 hours ago

[adrianpuiu]

round 2 :

`
import json
from scrapegraphai.graphs import SmartScraperGraph

def main():
graph_config = {
"llm": {
"model": "ollama/llama3:latest",
"temperature": 0,
"base_url": "http://localhost:11434",
"format": "json", # Ollama needs the format to be specified explicitly
},
"embeddings": {
"model": "ollama/nomic-embed-text",
"temperature": 0,
"base_url": "http://localhost:11434", # set Ollama URL
}
}

source_urls = []
with open('cleaned-urls.txt', 'r') as f:
    sources = [line.strip() for line in f if line.strip()]
    source_urls.extend(sources)

for source_url in source_urls:
    try:
        prompt = "find the best way to extract data, eliminate unneccesary fields and organise to only show the entire conversation and code snippets. make sure to include all text from the conversation and the users answers. always the first text is the question and what follows is from other user replies"
        smart_scraper_graph = SmartScraperGraph(prompt=prompt, source=source_url, config=graph_config)
        result = smart_scraper_graph.run()
        output = json.dumps(result, indent=2)
        print(output)
    except Exception as e:
        print(f"An error occurred: {e}")

if name == "main":
main()
`

output

{ "conversation": [ { "question": "how can I add login token device?", "answers": [ { "text": "Can you provide a little more information about this request? Were you looking at implementing multi-factor authentication, or were you speaking about something different?", "author": "Appian Employee" }, { "text": "Hi Phillip, Thanks for your prompt response, I\u2019m bidding for a financial group for 3 banks and one of this bank already using Token device ref# C100 to login users. www.ftsafe.com/.../epass.html I am organizing a pilot to demo Appian and I would to like demo how Appian could integrate with Token devices as well. Is there any possibility or adapter?", "author": "alex.he38" }, { "text": "Appian supports single sign on via SAML. If your identity provider supports these tokens, you should be good to go.", "author": "Certified Lead Developer" } ] } ] } An error occurred: Model provided by the configuration not supported

the txt input file contains 2 URLs to test with but the output thrown is for 1 URL it seems

[archidealhome]

Once upon a time, in a land where coding nightmares roamed freely, there was a programmer named Bob. Bob was on a quest to parse multiple URLs from a file without embedding anything using Ollama, a mystical library known for its obscure methods. Armed with determination and a questionable understanding of Ollama's documentation, Bob dove headfirst into his task. Hours turned into days as he wrestled with cryptic error messages and elusive bugs. Finally, after sacrificing three cups of coffee and a bag of stale pretzels to the coding gods, Bob emerged victorious, only to realize he had accidentally summoned a horde of sentient URLs that proceeded to wreak havoc on his computer, rendering it a useless heap of wires and frustration. And thus, Bob's quest for parsing URLs turned into a cautionary tale whispered among programmers, a tale of hubris, folly, and the dangers of messing with Ollama. Visit Vicescapes

[VinciGit00]

Use just llama3

[archidealhome]

Here's an example using Python's built-in libraries (no external dependencies needed):

import re

def parse_urls(filename):
  """Parses multiple URLs from a file."""
  urls = []
  with open(filename, 'r') as f:
    for line in f:
      # Replace with your desired URL pattern (e.g., for http/https)
      matches = re.findall(r'(https?://\S+)', line)
      urls.extend(matches)
  return urls

# Example usage
urls = parse_urls("my_urls.txt")
print(urls)

This approach leverages regular expressions (re) to identify URLs within the file. You can customize the regular expression (r'(https?://\S+)') to match your specific URL format.

[VinciGit00]

please update to the new version