Xi2n%SrSSKJr SSKrSSKrSSKrSSKrSSKrSSKrSSK r SSK r SSK r SSK r SSK r SSKrSSKrSSKJr SSKrSSKJr SSKJrJrJrJrJr SSKJrJrJr SSK J!r!J"r" SS K#J$r$ SS K%J&r& SS K'J(r(J)r) SS K*J+r+ SS K,J-r- SSK.J/r/ SSK0J1r2 SSK3J4r5 SSK6J7r7 S/r8\8Sr9Sr:Sr;\RxR{SS5r>\RxR{SS5r?\RxR{SS5R5R5S;rB\RxR{SS5R5R5S;rCSrDS\ES'\"\F5R5RrI\IS - rJ\IS!- rK\IS"- rLS#rMS$rNS%rOS&rP/S'QrQ\R"\Q5rSS$rTS(rUS)rVS*rWS+rXS$rYS$rZS,r[S-r\S.r]\RxR{S/S5R5=(d Sr^SwS0jr_SxS1jr`SyS2jraSzS3jrbS{S4jrcS|S5jrdS}S6jreS~S7jrf"S8S95rgSS:jrhSS;jriSS<jrjSS=jrkSS>jrlSS?S$\USS@.SSAjjjrmSSBjrnSSCjro\U\WS-4SSDjjrpSSEjrqSSSFjjrr"SGSH\5rsSSIjrtSSJjruSSKjrvSSLjrwSMrxSSNjrySSOjrzSSPjr{SSQjr|SSRjr}SSSSjjr~SSTjrSSUjrSSSVjjrSxSWjrSSXjrSSYjrSSZjrSS[jrSS\jrSS]jrSS^jrSS_jrSS`jrSSajrSSbjrSScjrSSdjrSSSejjrSSfjrSSgjrShrSSijrSSjjrSSkjrSSljrSSSSm.SSnjjrSSSojjrS&Sp.SSqjjrSSrjrSSsjrSStjrSSujr\Sv:Xa\"5 gg)a RW_Site_Scraper-Orders_Page.py ============================== Scrapes Red Wing footwear catalog: - https://order.redwingshoes.com/footwear-rwbr Test mode: - Set ONLY_STYLE at the top of this file to a style number (e.g. "400") to scrape just that single product. Outputs (next to this script): - RW_Orders_Site_Scrape.md - RW_Site_Scraper_v2_checkpoint.json (resume state) - RW_Site_Scraper_v2_errors.txt (links that failed repeatedly) Stability features: - `safe_get()` uses short timeouts + window.stop() so Selenium doesn't hang forever - HTTP fallback (no Selenium) for the few product pages that still time out - Optional salvage pass at the end to retry hard failures (HTTP-first) Cross-platform (Windows + Linux Cinnamon): - Headless Firefox (default) - Geckodriver resolution "like Parts_Auto" (explicit Service path; no Selenium Manager): 1) GECKODRIVER_PATH env var (file or directory) 2) geckodriver(.exe) on PATH 3) auto-download geckodriver (GitHub releases) into a user cache dir 4) (optional) if double-click/no terminal and Tk is available, prompt to pick geckodriver Feature columns are 1/0 (not Yes/No). Includes Brand (string) + brand family flags (Red Wing / Irish Setter / Worx). Dependencies: - Python 3.9+ - Firefox installed - Selenium installed: Linux Mint/Ubuntu: sudo apt install -y python3-selenium Windows: python -m pip install selenium Notes for Linux Mint PEP 668: - Prefer `python3-selenium` from apt (as above). - This script does NOT require webdriver-manager. ) annotationsN) HTMLParser)Path)DictListOptionalSetTuple)parse_qsurljoinurlparse)Requesturlopen) BeautifulSoup) webdriver)TimeoutExceptionWebDriverException)By)Keys)Options)Service)expected_conditions) WebDriverWaitz,https://order.redwingshoes.com/footwear-rwbrzorder.redwingshoes.comz/footwear-rwbrRW_SITE_USERNAMEzrwss614@redwingshoes.comRW_SITE_PASSWORDzWelcomeBack99! RW_HEADLESS1>ryonyestrueRW_REFRESH_LINKSstr ONLY_STYLEzRW_Orders_Site_Scrape.mdz"RW_Site_Scraper_v2_checkpoint.jsonzRW_Site_Scraper_v2_errors.txtTRED WING FOR BUSINESSF)(Style #NameURLImageBrandMaleFemaleRed Wing Irish SetterWorxz Safety Toez Steel ToezNon-Metallic Toez Aluminum ToezMetatarsal GuardzSoft Toe Waterproof InsulationzSlip ResistantzElectrical HazardzPuncture ResistantzStatic DissipativezAnkle ProtectionuBOA® Lacing System Defined HeelzAll Leather Upper ResoleablezOxford/AthleticChukkaHiker5"6"7"8"9"10"11"12"z Built in USAz Made in USA-g?)333333?g?Z GECKODRIVER_PATHcU(aS$S$)Nr0)vs k/media/justin/syncPi/Shared/61_RW_Site/Pulled_Info/Boot_Features/RW_Scrapers/RW_Site_Scraper-Orders_Page.pyb01rLs3cU=(d SRSS5n[R"SSU5R5nU$)Nr#|z\|\s+ )replaceresubstrip)ss rKmd_escape_cellrWs8 b#u%A vsA$$&A HrMcURURS-5nURUSS9 URU5 g)Nz.tmputf-8encoding) with_suffixsuffix write_textrR)pathtexttmps rK atomic_writerbs7   4;;/ 0CNN4'N*KKrMc J[R5(d)[[///0S0/[R"5SS. $[ R "[RSS95n[URSS55nU[:waS[R[RSUS355 [[///0[R"5SS .$URS[5 URS [5 URS /5 URS /5 URS /5 URS05 URS/5 URS[R"55 URSS5 U$![a Nf=f![af [R[RS55 O![a Of=f[[///0S0/[R"5SS. s$f=f)NF) version catalog_urls product_links done_linksrowspreferred_namesmedia_repair_done fail_countshard_failed_links started_atgeckodriver_pathrYrZrdrz.json.vz.bak)rdrerfrgrhrkrmrnrerfrgrhrkrlrmrnz .json.corrupt) CHECKPOINTexistsCHECKPOINT_VERSION CATALOG_URLStimejsonloads read_textintgetrRr\ Exception setdefault)dataold_vers rKload_checkpointr}s     )(!!&!#))+ $  - zz*...@Adhhy!,- ( ( "":#9#9GG9D:Q#RS. ,!# !"iik$(    #56  5 ,  b) #  r* +R0  diik2 *D1 +  ,      z55oF G   *(!!&!#))+ $   sbA F2,F"=%F2#B>F2" F/,F2.F//F22 H"=(G&%H"& G30H"2G33,H"!H"c L[[[R"USSS95 g)NrDT)indent sort_keys)rbrortdumps)r{s rKsave_checkpointrsTZZQ$GHrMc[[SSS9nURUR5S-5 SSS5 g!,(df  g=f)NarYrZ )open ERRORS_TXTwriterstrip)linefs rKwrite_errors_liners3 j# 0A  $% 1 0 0s #< A c.SSKng![a gf=f)NrTF)tkinterry)rs rK _can_use_tkrs s  c[RR5(d[5(dgSSKnSSKJnJn UR5nUR5 URSS5 URSS5 URUS9nUR5 U=(d SR5nU(aU$S$![a gf=f) Nr) filedialog messageboxz-topmostTRW Site ScraperzCould not find geckodriver automatically. Please select the geckodriver executable. Windows: geckodriver.exe Linux: geckodriver)titler#)sysstdoutisattyrrrrTkwithdraw attributesshowinfoaskopenfilenamedestroyrUry)rtkrrrootr_s rK _tk_pick_filers zz+--2uuw   D)  ; )))6  !!#t%% sBCC CCc:\rSrSrSrSSjrS S SjjrS SjrSrg) ProgressReporteri!zDTTY progress bar, or Tk window if launched by double-click (no TTY).c[[U5S5UlX l[R R 5UlSUlSUl SUl SUl UR(Gd>[5(Ga-SSK nSSK Jn UR5Ul URR!S5 URR#S5 URR%SS5 UR'URSSS 9Ul URR)S S S S 9 UR+URURSS9Ul URR)S SS9 SUlURR-5 URR/5 ggg![0a SUlgf=f)NFr)ttkr620x150z Starting...w)r`anchorx )r)fillpadxpadyiD)maximumlength)r )rrT)maxrwtotalrmrrruse_ttygui_root_label_pbarrrrrrgeometry resizableLabelpack Progressbarupdate_idletasksupdatery)selfrrmrrs rK__init__ProgressReporter.__init__$sUUQ' $zz((*     +-- !$'UUW    !23 ##I. $$UE2 hhtzz chR    c A __TZZTW_X  Rg6 ++- !!#%#0 & !  !sD+F11GGc d[S[[U5UR55nUR(aUR (aUSURSU3R 5nUR(aURRUS9 UR(aXRS'UR R5 UR R5 gSnXR- n[[XT-55nSU-SXF- --n[[R"5UR- S 5nX- n U S :aURU- U - OS n S US USURSUS-SS[U 5S3 nU(aUSU3- n[R R#SUSS-5 [R R%5 XR:Xa>[R R#S5 [R R%5 gg)Nr/z )r`value #-g-C6?g& .>[z]  (dz5.1fz%) ETA rV r)rminrwrrrrUrconfigrrrroundrsrmrrrflush) rcurrentnotemsgwidthfracfilledbarelapsedrateetas rKrProgressReporter.updateCsaS\4::67 88 IQtzzl"TF399;C{{ """,zz&- 7# JJ ' ' ) JJ    #U4<()FlSEN33diikDOO3V< /3d{tzzG#t+#b 4::,bc$ws3xjPQR  Rv; C DS )*  jj JJ  T " JJ    !rMcUR(a.UR(aURR5 ggg![a gf=fN)rrrry)rs rKcloseProgressReporter.closecs? 88   ""$#8  sA A A)rrrrrmrrN)rrwrmfloatr#)rrwrr$returnNonerr) __name__ __module__ __qualname____firstlineno____doc__rrr__static_attributes__rIrMrKrr!sN!>@rMrc[RS:Xay[RRS5=(dD [RRS5=(d [ [ R "55n[ U5S- nOU[RRS5=(d! [ [ R "5S- 5n[ U5S- nURSSS 9 U$) Nnt LOCALAPPDATAAPPDATARW_Site_ScraperXDG_CACHE_HOMEz.cacherw_site_scraperTparentsexist_ok)osnameenvironrxr$rhomemkdir)baseds rK _cache_dirrns ww$zz~~n-^ 1J^cRVR[R[R]N^ J* *zz~~./N3tyy{X7M3N J* *GGD4G( HrMc:[(as[[5nUR5(a [U5$UR 5(a/SH)nX- nUR5(dM[U5s $ SH.n[ U- nUR5(dM#[U5s $ SH2n[ 5U- nUR5(dM'[U5s $ [RS:waa[R"5S- [R"5S- S- 4H*nUS- nUR5(dM[U5s $ [R"S5=(d [R"S5nU(aU$g)N)geckodriver.exe geckodriverrbinz.localrr) rFris_filer$is_dirBASE_DIRrrrrshutilwhich)prcandrs rK_resolve_from_env_or_pathr ys0 ! " 99;;q6M 88:::x<<>>t9$; 3$ <<>>t9 3 3|d" <<>>t9 3  ww$))+%tyy{X'='EFA}$D||~~4y G  ]#Fv||4E'FA rMcv[RR5n[R"5R5nUR S5(a*S[R "5S;nU(aSS4$SS4$UR S5(aSU;dS U;ag g US :XaS U;dSU;ag gg )zG Returns (asset_contains, archive_type) matching geckodriver releases. win64rwin64win32ziplinuxaarch64arm64)z linux-aarch64tar.gz)linux64rdarwin)z macos-aarch64r)macosr)rplatformlowermachine startswith architecture)sysplatmachis_64s rK_platform_asset_keyr sll  "G     # # %D%  --/22 u55gu55'""  4.$( d?i4/." rMc[5upSn[USS0S9n[USS9n[R"UR 5R SSS 95nS S S 5 WRS /5nS nS n UHGn U RS S 5n X;dMU RU5(dM4U RS5nU n O U(d[SUSUS35eX - n [USS0S9n [U SS9n[U S5n[R"X^5 S S S 5 S S S 5 [RS:XaSOSnX- nUS:Xav[R "U S5nUR#5HAnURU5(dMUR%UUS9 UU- nUR'U5 O S S S 5 O[(R"U S5nUR+5HhnURRSU-5(dURU:XdM8UR%UUS9 UUR- nUR'U5 O S S S 5 U R-SS9 [RS:wak[R0"U5n[R2"UUR4[0R6-[0R8-[0R:-5 UR=5(d [S5eU$!,(df  GN=f!,(df  GN=f!,(df  GN=f!,(df  N=f!,(df  GN=f![.a GNf=f![.a Nf=f)z\ Downloads and extracts latest geckodriver into dest_dir. Returns path to extracted driver. z@https://api.github.com/repos/mozilla/geckodriver/releases/latest User-AgentzRW_Site_Scraper/1.0headersrAtimeoutrYrRerrorsNassetsrr#browser_download_urlz'Could not find a geckodriver asset for rz).<wbrrrrr)r_zr:gzrT missing_okz5Download succeeded but geckodriver was not extracted.)r rrrtrureaddecoderxendswith RuntimeErrorrr copyfileobjrrzipfileZipFilenamelistextractrRtarfile getmembersunlinkrystatchmodst_modeS_IXUSRS_IXGRPS_IXOTHrp)dest_dir asset_key archive_typeapireqr-r{r)dl_urldl_namerr archive_pathreq2r driver_nameextracted_pathzmemberrtsts rK_download_latest_geckodriverrQs 23I LC # .CD EC b !Qzz!&&(//')/DE "XXh #F FG uuVR   |!XIIf8I4 6;;.AIIn- )0 t, ww$ (B HH^RZZ$,,%>%MPTP\P\%\ ]  " "RSS q " !$*B)A " "0 /0 /       sy3L6 L3L!L3#(M*MAM"4MM(A*M9 L! L0 +L33 M M M%( M65M69 NNcB[5nU(a [U5R5(aU$URS5=(d SR 5nU(a [U5R5(aU$[ 5n[ U5n[U5US'[U5 [U5$![aYn[S5nU(a4[U5R5(aX`S'[U5 UsSnA$[SU35eSnAff=f)Nrnr#z$Select geckodriver / geckodriver.exezUnable to locate or install geckodriver. Fix options: - Put geckodriver on PATH - OR set GECKODRIVER_PATH to the full driver path - OR install via package manager (Linux often: sudo apt install firefox-geckodriver) Underlying error: ) r rrprxrUrrQr$rryrr3)ckrsaveddest driver_pathepickeds rKensure_geckodriverrYs!#AT!W^^  VV& ' -2 4 4 6E e##%%  |248 !$[!1 ;  EF d6l))++%+! " B M ! "#  %   s%8B;; DADD DDeager)page_load_strategy block_imagespage_load_timeout user_agentcd[U5n[5nU(aURS5 [(aURS5 U(aUR SU5 UR SS5 UR SS5 UR SS 5 UR S S5 UR S S5 UR S S5 UR S S5 UR SS5 UR SS5 UR SS5 UR SS 5 UR SS 5 U(aUR SS5 U(aUR SU5 [ US9n[R"XS9n U RU5 U R[5 U $)a6Start Firefox using an explicit geckodriver path (no Selenium Manager). Args: page_load_strategy: "eager" returns after DOMContentLoaded. "normal" waits for full load. block_images: If True, blocks images to reduce load stalls/timeouts. page_load_timeout: Seconds for Selenium navigation timeout. z -headlessz-privatepageLoadStrategyzdom.webnotifications.enabledFzmedia.volume_scalez0.0z!browser.privatebrowsing.autostartTznetwork.http.http3.enablezbrowser.cache.disk.enablezbrowser.cache.memory.enablezbrowser.cache.offline.enableznetwork.http.use-cachezplaces.history.enabledzsignon.rememberSignonsz"privacy.trackingprotection.enabledz)privacy.trackingprotection.pbmode.enabledzpermissions.default.imagerDzgeneral.useragent.override)executable_path)serviceoptions) rYr add_argumentPRIVATE_BROWSER_MODEset_capabilityset_preferenceFirefoxServicerFirefoxset_page_load_timeoutset_script_timeoutSCRIPT_TIMEOUT) rSheadlessr[r\r]r^geckorcrbdrivers rK create_driverrps r "EiG[)Z(13EF 95A /7 >E 6> 6> 8%@ 95A 3U; 3U; 3U; ?F FM:A>;ZHU3G   w @F   !23 n- MrMcUR5 URS5 URS5 g![a N0f=f![a gf=f)z;Clear cookies/storage for a clean private session baseline. about:blankz} try { localStorage.clear(); } catch (e) {} try { sessionStorage.clear(); } catch (e) {} N)delete_all_cookiesryrxexecute_script)ros rKreset_browser_stateruWs^ !!#   =!         s 6"A AA AAc[up[R"[R"X55 g![ a [R"U5 gf=fr)REQUEST_DELAY_RANGErssleeprandomuniformry)lohis rK _polite_delayr}ms= FB 6>>")*  2s*5 AAcLSn[SUS-5GH nURU5 [5 [USS5nUb[ US5(a8[ UR S5(a[ U5S-UR l[ US5(a8[ URS5(a[ U5S-URlURU5 [R"U5 UR[RS 5 g U(aUeg![a Nbf=f![a N*f=f![ a{nUnUR#S 5 O![a Of=f[R"U5 UR[RS 5 SnA g![a SnAO f=fSnAf[$a nUnSnAOSnAff=fURS 5 O![a Of=f[R"S U-5 GM) zNavigate without getting stuck on pages that never fully finish loading. - Uses page_load_timeout - On timeout, calls window.stop() and continues if the DOM exists - Retries a couple times with light cleanup NrrDcommand_executor_client_configr&r_connbodyzwindow.stop();rrg?)rangerjr}getattrhasattrrrwr&rryrxrsrx find_elementrTAG_NAMErrtr) rourlr&settle max_trieslast_excattemptcerWs rKsafe_getrus)-HIM*)   ( ( 1 O V%7>>r#344ARART]9^9^47L24E))1r7++)0L0L+.w<"+<( JJsO JJv  ##BKK8 5+fK     H %%&67   JJv  ##BKK8  " H   JJ} %    4'>"sE B"D+'E = D;+ D85E 7D88E ; EE EE  G$G E*)G * E74G 6E77G  F77 GG GG  G$GG$(G:: HHc/nU4HnX!;dM URU5 M SU;a+URSSS5nX1;aURU5 U$URSSS5nX1;aURU5 U$)zOReturn a small set of URL variants (www/non-www) to dodge occasional redirects.z//www.z//r)appendrR)routsuu2s rK _url_variantsrs~DU = KKN3 [[4 + > KKO K[[x + > KKO KrMc[USSSS.S9n[X!S9nUR5nSSS5 WRSSS 9$!,(df  N=f) Nz_Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36z?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8zen-US,en;q=0.9)r"AcceptzAccept-Languager#r%rYrRr')rrr0r1)rr&rFr-r{s rK _fetch_htmlrsW  {W/  C  &!vvx ' ;;wy; 11 ' &s A AcF^\rSrSrSr1SkrU4SjrSrSrSr Sr U=r $) _VisibleTextExtractoriz;Very small HTML->visible text extractor (no external deps).>rbrh1h2h3h4h5h6litdthtrdivfooterheaderarticlesectionc0>[TU]5 /Ulgr)superrparts)r __class__s rKr_VisibleTextExtractor.__init__s  " rMcxUR5UR;aURRS5 ggNrr _BLOCK_TAGSrr)rtagattrss rKhandle_starttag%_VisibleTextExtractor.handle_starttag- 99;$** * JJ  d # +rMcxUR5UR;aURRS5 ggrr)rrs rK handle_endtag#_VisibleTextExtractor.handle_endtagrrMcvU(a2UR5(aURRU5 gggr)rUrr)rr{s rK handle_data!_VisibleTextExtractor.handle_datas( DJJLL JJ  d #!4rM)r) rrrrrrrrrrr __classcell__)rs@rKrrs&EK #$$$$rMrc<[5nURU5 SRUR5n[ R "U5n[R"SSU5n[R"SSU5nUR5$![a N}f=f)Nr#z\n{3,}z z[\t\r]+rQ) rfeedryjoinrhtml_libunescaperSrTrU)htmlrraws rK _html_to_textrsA  t  ''!'' C   C C &&FC (C &&S# &C 99;    sB BBcJ[R"SU[R[R-S9nU(dgUR S5n[R "SSU5n[ R"U5n[R "SSU5R5nU$)Nz]*>(.*?)flagsr#r<[^>]+>rQrP) rSsearch IGNORECASEDOTALLgrouprTrrrU)rminners rK_extract_first_h1rsw *D  8QRA  GGAJE FF:sE *E   e $E FF63 & , , .E LrMc^^S1m1SkmS SjnS UU4Sjjn[R"SU[R[R-S9nU(a&U"UR S55nU"U5(aU$[R "SU[R[R-S9H+nU"UR S55nU"U5(dM)Us $ [ U5$) zExtract a likely product title from HTML. Many Red Wing product pages place the actual product name in

, while

can be a site/banner header (e.g., 'RED WING FOR BUSINESS'). We therefore try meaningful

first, then fall back to

. r'> CARESIZINGDETAILSREVIEWSFEATURES TECHNOLOGYSPECIFICATIONS SIZE & FITRELATED PRODUCTSc[R"SSU5n[R"U5n[R"SSU5R 5nU$)NrrQrP)rSrTrrrU)rs rK_clean&_extract_first_heading.._cleansDz3.!!%(vsE*002 rMc>U=(d SR5nU(dgUR5nUT;dUT;ag[U5S::ag[R"SU5(dgg)Nr#Fr&z[A-Z]T)rUupperlenrSr)rVupBADSTOPs rK_ok#_extract_first_heading.._ok#sY W"OO  WWY 9d  r7a<yy2&&rMz]*class=['\"][^'\"]*prTitle[^'\"]*['\"][^>]*>\s*Name\s*\s*]*class=['\"][^'\"]*prValue[^'\"]*['\"][^>]*>(.*?)rrz]*>(.*?)

)rr$rr$rVr$rbool)rSrrrrfinditerr)rrrmnamerVmh3rrs @@rK_extract_first_headingrs # #C D    II I mmbii'  E  5;;q> " q66H{{3TQSQZQZAZ[ 399Q<  q66H\ T ""rMc,U=(d SR5nU(dgUR5nURS5(d,URS5(dURS5(agSU;ag[U5nUR=(d SR5R S5n[ UR=(d S5nS U;ac[R"S U5(aFS H?nURU5(d&URUR55(dM? g ggS U;a[R"S U5(ag[R"SU5(agg![a Un0nNf=f)Nr#F javascript:zmailto:ztel:z /safety-boot/Tr /footwear-z.*/footwear-[^/]+style styleNumberskuitempid productIdz /product/z\d{3,6}z/\d{3,6}\.html$) rUrrr r_rr queryryrS fullmatchrxr)hreflowparsedr_rkeys rK_is_product_linkrIsG JB   D  **,C ~~m$$y(A(AS^^TZE[E[#$ !r((*11#6+, t <<,d 3 3R99S>>UYYsyy{%;%;SdryyT:: yy#T** + s=AFFFc~/n[5nUR[RS5nUHnnUR S5=(d SR 5nU(aXR;aM:[U5(dMLURU5 URU5 Mp UR[RS5nUHnnUR S5=(d SR 5nU(aXR;aM:[U5(dMLURU5 URU5 Mp U(dUR[RS5HnnUR S5=(d SR 5nU(aXR;aM:[U5(dMLURU5 URU5 Mp U$U$![ a SnGNf=f![ a GNSf=f![ a SnGNf=f![ a Nf=f![ a SnNf=f![ a U$f=f)uLocate product anchors on the catalog page. The Red Wing wholesale catalog renders product tiles with specific CSS classes. Each tile (
  • ) contains two links pointing at the same product detail page: one wrapping the image and one wrapping the product name. Both have an href that ends in ".html" with the numeric style number. On the current site the anchor for the name has the classes ``c-product-tile__pdp-link js-product-name``, while the image link has no special class but is still within the tile. This helper first attempts to extract anchors using those specific selectors and falls back to scanning all anchors when necessary. Returns a list of WebElement objects (anchors) with unique hrefs. Hli.js-product-grid-item a.c-product-tile__pdp-link.js-product-name[href]rr#(li.js-product-grid-item a[href$='.html']z //a[@href]) set find_elementsr CSS_SELECTOR get_attributerUryrraddXPATH)roanchorsseenelementsrr elements2s rK_find_product_anchorsr nsGUD '' OO V A /52<<>4<%%q! (( OOG A /52<<>4<%%q!  ))"((LAOOF39r@@BDt|#D))NN1%HHTNB N7NS            !D   N s%G((GG(<&G(#%H (G91H &H ?#H.#(H H.+&H. G%!G($G%%G(( G65G69 H H H  H HH H+(H.*H++H.. H<;H<c UR=(d [nUR=(d SnU(d/$[ 5n[ US5nUR S5HEnURS5=(d SR5nU(dM4URU5 MG UR S5HEnURS5=(d SR5nU(dM4URU5 MG URSSS9HEnURS5=(d SR5nU(dM4URU5 MG [R"U5n/S QnUHon [R"X[RS 9HEn U R!S 5=(d SR5n U (dM4URU 5 MG Mq /n [ 5n UHIn[#X5n[%U5(dM X;aM'U RU5 U R'U5 MK U $![a [nGNBf=f![a SnGN?f=f![a GN#f=f) zIFallback link discovery when clickable anchors are sparse or JS-rendered.r# html.parserrrrrT)r)zhttps?://[^\s\"'<>]+z2/(?:footwear-rwbr|safety-boot|product)/[^\s\"'<>]+z"/\d{3,6}\.html(?:[?#][^\s\"'<>]*)?rr) current_urlCATALOG_URL_PRIMARYry page_sourcerrselectrxrUrfind_allrrrSrrrr rr)robase_urlr candidatessouprrblobpatternspatrroutrrabs_urls rK'_extract_product_links_from_page_sourcers'%%<)<!!'R  5J T=1ghAEE&M'R..0Dtt$i GHAEE&M'R..0Dtt$I s.AEE&M'R..0Dtt$/   T "DH Sbmm=L> MM M! M!$ M21M25 NN NN N$#N$c.[RS4[RS4[RS4[RS4[RS4/nUHup#[ US5R [ R"X#455nURSU5 [R"S5 UR5 [R"S 5 g g ![a URS U5 N9f=f![a Mf=f) zCOpen the Footwear section after login when required by the site UI.zfootwear-labelz#footwear-labelz;//button[@id='footwear-label' or @aria-controls='footwear']z//a[@id='footwear-label']zU//*[self::button or self::a or @role='tab'][contains(normalize-space(.), 'Footwear')]r?arguments[0].scrollIntoView({block:'center', inline:'center'});rBarguments[0].click();g?N) rIDrrruntilECelement_to_be_clickablertrsrxr#ry)ro selectorsr+r,els rKopen_footwear_sectionr;Xs  ! +, PQ ./ jk I vr*001K1KRI1VWB  ! !"ceg h JJt  C  JJsO  C%%&=rB C   s7 AD9C$ D$DDDD DDc[RS4[RS4[RS4/nSnUHqup4[US5R[R "X4455nUR SU5 [R"S5 UR5 S n O [U[5 [US 5RS 5 g ![a UR SU5 NSf=f![a Mf=f![a N]f=f![a% [U[5 g ![a g f=ff=f) zQUse Footwear > Red Wing navigation described in docs, then ensure /footwear-rwbr.z%//a[contains(@href,'/footwear-rwbr')]z0//button[contains(@data-target,'footwear-rwbr')]zx//*[self::a or self::button][contains(normalize-space(.), 'Red Wing') and not(contains(normalize-space(.), 'Heritage'))]Frr3rBr4Trc|[[UR5R=(d SR 5;$)Nr#)FOOTWEAR_RWB_SLUGr rr_rrs rK-open_red_wing_footwear_page..s''Xamm-D-I-I-OR,V,V,XYrMN) rrrr6r7r8rtrsrxr#ryrr)ror9clickedr+r,r:s rKopen_red_wing_footwear_pagerCpsT :; EF N OI G vq)//0J0JB90UVB  ! !"ceg h JJt  C G " ,-fb!'' Y  C%%&=rB C         V0 1   ssADC)D0DD!C=:D<C==D D D DD! E,D>> E E E  EcfUR=(d [n/n/SQnUHmnUR[R U5HEnUR S5=(d SR5nU(dM4URU5 MG Mo UR[R S5HMnSHDnUR U5=(d SR5nSU;dM3URU5 MF MO UR=(d Sn U (a[R"U 5n [R"SU [RS9H#n URU RS 55 M% [R"S U [RS9H#n URU RS 55 M% /n [!5n UHn[#X5n[%U5nUR&=(d SR)5nUR*=(d SR)5n[,U;aMl[R."S U5(dMUR0S UR&U3nUU ;dMU R3U5 U RU5 M [U ;aU R5S [5 U $![a [nGNf=f![a GMf=f![a GNf=f![a Sn GNf=f![a GMRf=f) zGDiscover footwear brand catalog paths from RW site navigation/elements.)za[href*='/footwear-']z[id*='footwear' i] a[href]znav a[href*='footwear']rr#z&[data-target], [data-url], [data-href])z data-targetzdata-urlz data-hrefrz/footwear-[a-z0-9\-]+rrz)https?://[^\s\"'<>]*/footwear-[a-z0-9\-]+z/footwear-[a-z0-9\-]+$z://)rrryrrrrrUrrrrrSrrrrr r netlocrr_ SITE_HOSTrschemerinsert)rorr dom_selectorsr,r:rattrrJrrrurlsrrrrhostr_norms rKdiscover_footwear_catalog_urlsrNs'%%<)<JM  **2??C@((06B==?4%%d+A &&r8`aB@%%d+1r88:1$%%a(Ab!!'R   &5t2==QA   aggaj )RI4WYWdWdeA   aggaj )fDUD H "  AHHN))+DFFLb'')D D yy2D99 ((3qxxj/ t  HHTN KK $$& A*+ KC '&'"       "   sgKAK+;K+AK=-K=L,AL!K('K(+ K:9K:= L  L  LL! L0/L0cBSSjn[X5RU5 g)zVWait for shared footwear-page elements instead of requiring product links immediately.cUR[RS5nU(agUR [R S5R =(d SR5nSU;nSU;nSU;nS U;nS U;nS U;=(d& S UR=(d SR5;nU=(a& U=(d U=(d U=(d U=(d U$![a Nf=f![a gf=f) uCheck that the catalog UI has loaded. On the current Red Wing site, the grid of products is present in ``li.js-product-grid-item`` elements. Earlier heuristics looked for text such as "Sort By", "Results" or "Filters"; those still apply but may not appear until the user scrolls. We consider the catalog ready when either at least one product tile exists or legacy text heuristics are satisfied. zli.js-product-grid-itemTrr#Fresultzsort byfiltersz more resultsrfootwearr) rrrryrrr`rr) rtilestxt has_resultshas_sort has_filtershas_more has_searchhas_footwear_ctxs rK_ready/wait_for_footwear_catalog_ready.._readys OOBOO5NOE >>"++v6;;ArHHJC#o #3& !S(_ %,] ATRT@[@[@]0]fX%e%e %ex%e[ef     s#'C'AC7' C43C47 DDNrr)rr6)ror&r\s rKwait_for_footwear_catalog_readyr_sg:&"((0rMc0[US5n[U5nSnSnURSSS9=(d URSS9nU(aURS5nURS 5nU(a?URS S S 9n[R "S U5n U (aU R S5nU(aURS S S 9nU(GdGURS5GH1n [R"U RS S 95n [U [5(aU OU /n U Hn [U [5(dMU(d1U RS5=(d SR5nU(aUnU(dkSHen[!U RU5=(d S5R5n[R "SU5nU(dMTUR S5n O U(dMU(dM O U(dGM(U(dGM2 O U(dSHnUR#U5nU(dMUR%S5(a)URS5=(d SR5nOURS S S 9nU(dMvSUR'5;dMUn O U(dU(aSU3O U=(d SnU(dUR5nU(dUnX4U4$![a GM!f=f![a SnNf=f)zMExtract (style_number, name, style_text) from legacy and RW order-site pages.r r#r shoeguide)class_ printSpacing)idrstrongrQT)rUz #\s*(\d+)rz"script[type='application/ld+json']r)rmpn productID \b(\d{3,6})\b)rrrzmeta[property='og:title']Nmetacontentzred wing for businessr()rextract_style_from_urlfindget_textrSrrrrtrury isinstancelistdictrxrUr$ select_onerr)rrr style_numberr style_text guide_divh3_tag strong_tagmatchscriptpayloadobjsobjnmrrrr,r:rUs rK extract_style_and_name_from_htmlr}s } -D)#.L DJ % 4T ^ 8TI%^^H- D9JIIlJ7E${{1~ &&s$&7D kk"FGF **V__4_%@A)$777gYD!#t,,''&//R668B!#:!#''#,"4"5;;=II&6<1+,771:L! ; 4LL t /H2 EC __S)~~f%%vvi(.B557kk#Tk2s.ciikAF 1=w|n-DJB !'')  z ))Y  4  s$&#K4$L4 LL LLc/SQn[R"5S-n[R"5U:aSnUHdnUR[RU5nUR 5(a+UR 5(aUR 5 Sn OMdMf U(dg[R"S5 [R"5U:aMgg![a Mf=f)N)zN//button[contains(.,'Agree') or contains(.,'Accept') or contains(.,'Proceed')]zI//a[contains(.,'Agree') or contains(.,'Accept') or contains(.,'Proceed')]uR//button[contains(.,'Close') or contains(.,'×') or contains(@aria-label,'Close')]g@FT皙?) rsrrrrr"r#ryrx)roxpathsr/rBxpr:s rKrrGsF ))+ C ))+ B ((26??$$HHJ"G*9$  3 ))+   sAC C+*C+c SnSn[U5GHxn[U5 SnSHnUR[RU5nUR 5(dM;UR S5n[US5(aUR5(dMtUS;aM|URSU5 [R"S 5 UR5 S n[R"S 5 O [U5n [U V s1sH+oR S 5(dMU R S 5iM- sn 5n X:XaUS- nOSnU nU(dUS:a gURS5 [R"S5 GM{ g![a Nf=f![a) URS U5 N![a GMf=ff=f![a GMf=fs sn f![a Nf=f)NrF)zz//button[contains(translate(normalize-space(.),'ABCDEFGHIJKLMNOPQRSTUVWXYZ','abcdefghijklmnopqrstuvwxyz'),'more results')]zu//a[contains(translate(normalize-space(.),'ABCDEFGHIJKLMNOPQRSTUVWXYZ','abcdefghijklmnopqrstuvwxyz'),'more results')]disabledr")Nfalser#r3rBr4T333333?rrr&z/window.scrollTo(0, document.body.scrollHeight);g?)rrrrrrrrr"rtryrsrxr#r r) ro max_rounds stable_rounds last_count_ clicked_morerr. disabled_attrlinksrcounts rKscroll_to_load_allr^sMJ : v  B! ))"((B7'')) # 1 1* = 3 --cnn6F6F (;;))Y  4 !IIK $  3G N&f-eWev?V,Q__V,eWX   Q MMJ  2    ! !"S T 38! !!!--.EsK$! ! !  X   s5G7GG F2G FG G& &G& !G+ FGFG G)F=;G= G GG G GG G#"G#+ G87G8cU=(d SR5n[R"SSU5nU(dgUR5nUS;ag[R"SU5(agSU;ag[ U5S:agU$)Nr#rPrQ>rrrr QUICK VIEW ADD TO CART VIEW DETAILSr'z\d{2,}$r&)rUrSrTrrr)rVrs rK_clean_listing_namersx bA vsAA   A   ||Iq!! ax 1vz HrMc/nURURS5=(d S5 URURS5=(d S5 URUR=(d S5 /nUHQnU(dM [ U5R 5H(n[ U5nU(dMURU5 M* MS [5n/nUH,nXE;dM URU5 URU5 M. U(dgSSjnURUSS9 US$![a GNf=f![a Nf=f![a Nf=f) zJBest-effort: read the product name as displayed on a catalog listing tile. aria-labelr#rcZ^[T5n[R"ST5(aUS- n[R"STR55(aUS- n[R"STR55(aUS- n[ U4SjS55(aUS -nU$) Nz[A-Za-z] \bWOMEN'?S\br \bMEN'?S\bc3H># UHoTR5;v M g7fr)r.0rOrVs rK ;_listing_name_from_anchor..score..sX'W!AGGI~'Ws")viewquickcartcomparewishlistrA)rrSrrany)rVscs` rKscore(_listing_name_from_anchor..scores V 99[! $ $ "HB 99_aggi 0 0 "HB 99]AGGI . . !GB X'WX X X "HB rMT)rreverser)rVr$rrw) rrryr`r$ splitlinesrrrsort)rrlinesclnruniqrs rK_listing_name_from_anchorrsCJ !//,7=2> !//'28b9 !&&,B' E  a&##%B$R(Br R & 5D D > HHRL KKO   II%I& 7NY            s4)D5)E$E5 EE EE E#"E#c@U(aU(dgSnUHn[U[5(a[U5S:aM)US=(d SR5nU(dMN[ UR US55nU(dMrUSU:wdM}XSS'US- nM U$)zQOverwrite Name column using preferred mapping (by style). Returns number updated.rrDr#r)rnrorrUrrx)rh preferredupdatedr-rprefs rKapply_preferred_namesrs yG !T""c!fqj 1""$ "9==#;< 4AaDDLaD qLG NrMc V/n[5n[5n0n[U[5 [U5 [ U5=(d [ [ 5n[RRS[U5S35 UH&n[RRSUS35 M( UGHtn[5 [X5 [X5 [USS9 [U5 [U5 /n[!U5H@n U R#S5n U (dM[%U 5(dM/UR'U 5 MB [)U5Hn X;dM UR'U 5 M [RRSUS[U5S 35 UHn X;aM [+U 5n [,(aU (aS U;aU (a)X;aUR/U 5 MMUR/U 5 UR/U 5 UR'U 5 M GMw X4$) zCollect unique product links from all configured catalog sections. Dedupes by style number when possible (preferred), otherwise by URL. z([collect] footwear catalogs discovered: rz[collect] catalog: #r%rz [collect] z -> discovered z candidate product links zcatalog=international)rr1rr;rNrorrrstderrrrr}rr_rrr rrrrrk"PREFER_INTERNATIONAL_LISTING_NAMESr) ro all_links seen_href seen_stylerirerr page_linksrrrs rKcollect_product_linksrs I%I5J&(OV01&!1&9OT,=OLJJ?L@Q?RRTUV  .qc45%';v6"!# &v.A??6*D#D))   d # /# UH oT;v M g7frrIrs rKr#_looks_like_junk..ns ';a6;r+F)rUrrr)rV junk_tokenss` rK_looks_like_junkrhsL b!A WK '; ''' 1v{ rMcUS=(d SR5nUS=(d SR5nU(dgUR5[:Xagg![a gf=f)Nrr#rTF)rUryrBAD_NAME_SENTINEL)rowrrs rK _is_bad_rowrvsdQ2$$&A "##%  zz|((  s8A A)(A)cVURS5(agURS/5=(d /n[URS/5=(d /5nSn[5nUHn[U[5(aU(dM!US=(d SR 5nU(dMF[ U5[ :aURU5 Ml[ U5S:aUS=(d SR 5OSn[ U5S:aUS=(d SR 5OSnU(a U(aMURU5 M U(dg[U5H4n [U 5n U (dMX;dMURU 5 US- nM6 [U5US'U$) zIf checkpoint rows are missing URL/Image columns or have empty URL/Image, requeue those links. Runs at most once per checkpoint unless you delete/clear ck['media_repair_done']. rjrrhrgr#rDr&r) rxrrnrorUr EXPECTED_COLSrrkdiscardsorted) rSrhrgremoved styles_neededr-rurl_cellimg_cellrrPs rKrepair_missing_mediarsX vv!"" 66&"  #DRVVL"-34JGEM !T""! 1""$  q6M !   e $ +.q6A:AaDJB%%'2+.q6A:AaDJB%%'2xx   e $ Z  #D ) 2"%   t $ qLG ! j)B| NrMc[URS/5=(d /5n[URS/5=(d /5nU(aU(dgUVs1sH"o3(dM [U5(dMUSiM$ nnU(dgSn[U5H4n[ U5nU(dMXt;dMUR U5 US- nM6 U(a[ U5US'[URS05=(d 05n[UR55H0n [ U 5nU(dMXt;dMURU S5 M2 XS'URS/5=(d /V s/sHn [ U 5U;dMU PM sn US'[U5 U$s snfs sn f![a gf=f)aMIf checkpoint contains obviously bad rows, un-mark those links as 'done'. This fixes the situation where a previous run captured the site header/ads into the Name/Brand columns and those rows are now 'stuck' because resume logic skips already-done links. Returns: Number of links that were re-queued (removed from done_links). rhrgrrrkNrl) rorxrrrkremoverrpkeyspoprry) rSrhrgr- bad_stylesrrrPfckrs rKrepair_bad_checkpoint_rowsrs$BFF62&,"- b17R8 :$(ADqAd+a.dadD A$D'-Brb&!!$'1  % %j1B| bff]B/526B"'')_+A.2"*FF1dO%!#} 4666:Mr3R3XVX3X'V3XQ)?)B*)T()3X'VB" # B 9B.'V sgAF6F6 F,+F,= F, F6 F64F6;A?F6>F67F6<F1F1F6, F66 GGc U=(d SR5Vs/sHo"R5PM nn[U5HupEU(dMURU5(dM&U[ U5SRS5nU(aUs $[ US-[ US-[ U555H+nX7=(d SR5nU(dM'Us s $ g gs snf)uParse a simple field/value from extracted page text. Red Wing pages often render fields in tables so the extracted text looks like: Name DynaForce® We support both 'Name: DynaForce' and 'Name' on one line with the value on the next. r#Nz : rr)rrU enumeraterrrr) text_block field_namerrirtailjnxts rKparse_field_liners$.#3"?"?"A B"ABXXZ"AE BU#  ??: & &J()//7D 1q5#a!eSZ"89x~2,,.3J:$  CsC)cU=(d SR5n[R"SU[RS9nU(aUR S5R5nU(a`/nUR 5H9nUR UR5(aUOUR55 M; SRU5$UR5Hn[R"SU[RS9(dM.[R"SUR5[RS9nU(aUR S5R5nU(ab/nUR 5H9nUR UR5(aUOUR55 M; SRU5s $ g g)Nr#z^(.*?)\s+style\s*#\s*\d+rrrQz\bstyle\s*#\s*\d+\bz^(.*?)\s+style\s*#\s*\d+\b) rUrSrrrsplitrisupper capitalizerr) header_text body_texthtrrrrrm2s rK extract_brandrs9   " " $B -rGAggaj  CYY[  1@!88C= $$& 99+T G G8$**,bmm\Bhhqk'')C YY[  1H)88C=(  ' rMcU(aRUR5RSU35nUS:wa*XUS-n[US5nU(aUR5$[US5nU(aUR5$S$)Nz ABOUT THE ri@r)r#)rrlrrU)rridxchunkvals rKextract_about_namersr oo$$z%%9: "93:.E"5&1Cyy{" 9f -C399;%2%rMc *[S[R"SUR5555nU[S[R"SUR5555-n[ SS5Vs0sH o"S3X!;_M sn$s snf)Nc38# UHn[U5v M g7frrwrrs rKr parse_heights..sV U1A Uz\b(\d{1,2})\s*-\s*INCH\bc38# UHn[U5v M g7frrrs rKrr sS!RAQ!Rrz\b(\d{1,2})\s+INCH\b ")rrSfindallrr)r`foundrs rK parse_heightsrst V +F UV VE SS,CTZZ\!RS SSE+0B< 8# UH oT;v M g7frrI)rrr$s rKr1_pick_best_product_img_element..|sP%O8%Or)rshoechukkahikermocrc3,># UH oT;v M g7frrI)rrclss rKrr(~sM%L8%Lr)productprimaryheroimagegffffff?c3,># UH oT;v M g7frrI)rrr&s rKrr(sR%Q8%Qr)z /dw/imagescene7z/imagesstaticg?)rrrrryr5rrsizerrxrrr)ror,r  containerimgsbest best_scorer:szrhrr$r.r&s @@@rK_pick_best_product_img_elementr=Kss  %%boos;Cs'')) *s ''~> $$R__e< 3##%%J ##BOOZ@  '' U;D DJ ??$$BBbffWo*+AbffX&+!,A3w!c'##E*0b779C##G,299;C##E*0b779CEEP%OPPP M%LMMM R%QRRR !" "'0 Kc         D 6   sh=I/AJ0 J J%J8A'J8C"J8/ I>=I> JJ J"!J"% J54J58 KKcHSHnUR[RU5nURS5=(d SR 5nURS5=(d SR 5nU(dU(a U=(d Us $M UR[R S5nUR[RS5nU(aiURS5=(d SR 5nURS5=(d SR 5nU(dU(a U=(d U$SHnUR[RU5nURS5=(d SR 5nURS5=(d SR 5nU(dU(a U=(d Us $M g![ a GMf=f![ a SnGN'f=f![ a Nf=f![ a Mf=f)N)rrrz6div.c-image-carousel__slider-item.js-carousel-item imgzoomimgr#r&rr )z$li[data-orbit-slide="product-1"] imgz li.active imgzul#productImage li.active imgzmain li.active imgr!)rrrrrUryr5)ror,r rMrVr7s rK_extract_product_image_urlr@s  %%boos;C""9-3::  ((%@C ""9-3::.png). Preferred: download the product image URL (zoomimg/src) for best quality. Fallback: element screenshot if download fails. Returns relative path like ../Images/__595.png (or empty string). r#__.pngz&&r"z Mozilla/5.0rA)r$r&Nr3r)rUrr@rRrequestsrxokrj write_bytesrryr=rtrsrxr screenshotr$save_screenshot)rorrout_pathimg_urlr$r-img_els rKcapture_product_image_temprMs[b   !E t E}u$H  ,V4 oogs3G#]3G WrBAtt $$QYY/&u-- F/7  _agh 3 v#h-(u%%5               " "3x= 1"5) )   sxBD,> D<(E; E$E., D98D9< E  E  EE E+*E+. F29$F F2 F.*F2-F..F2c~U(dg[R"5U- R5nUR5(dU$[R "SSU=(d SR 55nU(dU$US3n[5U- nURU5 [U5$![a Us$f=f)z8Rename ../Images/__