Lok Ma Chau

Duties:

1. Design an efficient and intelligent crawler system that meets the requirements of large models, and perform multi-source (text, attachments, audio, video) data collection, processing and storage.

2. Optimize spider data format and quality according to the large model data preprocessing requirements.

3. Promote the construction of data collection platformization, achieve automation, distribution, and high concurrency of data collection and management.

4. Follow up on the latest anti-scraping technology and risk control strategy, and continuously optimize the crawler architecture and performance.

Requirements:

1. Bachelor's degree or above, majoring in Computer Science, Artificial Intelligence, Network Security, etc.

2. More than 3 years of experience in crawling and reverse engineering, those who have large model or AI-related field crawling application experience are preferred.

3. Familiar with and able to bypass common anti-scraping techniques (such as dynamic page loading, CAPTCHAs, JS encryption, account suspension, etc.).

4. Proficient in Python or other programming languages, familiar with crawling frameworks and automated collection technologies (such as Scrapy, Selenium, Playwright, DirssionPage, etc.).

5. Familiar with large model training and data processing pipelines, able to adjust data retrieval strategies according to model requirements, and efficiently extract and clean complex data.

6. Have entrepreneurial passion, strong self-motivation, good stress resistance, and rapid adaptation to changes. Actively and independently solve problems. Have good communication skills and team spirit.

1. Have large-scale data crawling and processing experience, especially for AI large model data collection and optimization.

2. Familiar with data encryption and decryption technology, able to crack complex encryption algorithms.

3. Have technical background in AI or large model field, able to understand and support model data needs.