fix: Respect enqueue_strategy after redirects in enqueue_links

[Mantisus]

Description

• Sets the enqueue_strategy attribute in Request during enqueue_links processing, for correct check of requests that have completed redirection.

Issues

• Closes: enqueue_links does not set the corresponding strategy in the Request.enqueue_strategy attribute #1606

[Pijukatel]

Hi, I am not sure what is the desired behavior here. You are overriding the strategy that is in the request.

Imagine we have (line from the test you added):

async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
...
await context.enqueue_links(requests=[Request.from_url(target_url, enqueue_strategy='same-hostname')], strategy='same-origin')

What is the desired behavior in such case? With your change, we will override the explicitly set enqueue_strategy from the request.

I am not really sure about the current behavior. I would expect all requests to inherit enqueue_strategy from the request that enqueued them (unless they explicitly pick a different strategy), but we are not doing that. We pick same-hostnameas the default. I would expect context.request.enqueue_strategy to be used as the default.
https://github.com/apify/crawlee-python/blob/master/src/crawlee/crawlers/_basic/_basic_crawler.py#L995

My point is that in this scenario, there are 3 sources of information for the enqueue_strategy and we have to understand how to handle their conflicts and which one of them is relevant in which case:

• context.request.enqueue_strategy
• enqueue_strategy (from newly created Request)
• strategy (from enqueue_links kwarg)

[Mantisus]

My point is that in this scenario, there are 3 sources of information for the enqueue_strategy and we have to understand how to handle their conflicts and which one of them is relevant in which case:

Excellent point. Thank you.

• strategy (from enqueue_links kwarg) - If the user has explicitly specified that all these requests should be set with this strategy. I think, it is priority.
• Request.enqueue_strategy - I'm not sure, because if these are user-generated requests, then the user can simply call add_requests. If the requests are obtained using extract_links and then passed to enqueue_links with different strategy parameters, then these are uncoordinated calls, and I would give priority to enqueue_links.
• context.request.enqueue_strategy - This is debatable, perhaps it would be worth doing when calling add_requests with a list of requests without enqueue_links?

@janbuchar @vdusek Perhaps you will have some ideas.

[vdusek]

Just typos, otherwise LGTM