New domain - robots.txt blocks all crawling in Google Search Console and Merchant Center

[Wages_of_sin]

Hello, for now the old domain still works ( https://shop.spreadshirt.***/SHOPNAME ), so for now I can still see my products on Google Search Console, and the feed works well on Google Merchant Center.

BUT, if I use the new domain ( https://SHOPNAME.myspreadshop.*** ), Google always shows an error saying that robots.txt is blocking the crawling. In Search Console I can’t manually index any page (with the old domain I still can), and Google Merchant Center can’t crawl a single product for the same kind of error (“page not crawlable due to robots.txt”, both mobile and desktop).

Now, I can still use the feeds from the old domain for the european shops in Merchant Center, but the USA, Canadian, and Australian shops have already disabled the old domain name. With those shops I’m forced to use the new domain, with the consequence that I can’t make anything being crawled, as Spreadshop’s robots.txt blocks every single page/product.

I alread sent 3 mails, but the final answer was “Unfortunately we cannot offer extensive and detailed IT-consulting for our shop platform.” and other totally unrelated answers.

I’m sure I’m not the only one with this issue, and I understand we’re in the middle of August, but can I just hope to have some kind of real answers about this as soon as possible? Thank you in advance. I hope this can be solved soon.

[nberl]

Same issue here!
Google told me to put a Link on 301 Request to navigate to the new domain … bäh imposibile.
If you change the domain please always do an previous check which features you are disableing.

look for help.

[Britta_Spreadshop]

Hey @Wages_of_sin,
you are absolutely right, but don’t worry: we are aware of that. We can only have indexing on one URL and since the new ones are just launched we decided to keep indexing the old URL’s until everybody is aware of the new ones. Please just have a little more patience.

[Adopteunlook]

Hello.
I have the same problem. All the changes are made, I can’t go back.
I’ll be patient, but no google ads are running anymore and so we don’t make any more money and we don’t make you earn either …
I hope this problem will be resolved quickly.
Cordially.

[Adopteunlook]

Hello Britta,

For 20 days, I have been broadcasting more google ads campaigns because of the change of domain. So no orders, no money!
Today, Google Merchant Center suspended my account (see image).

When will your teams switch to the new domain for google merchent center, it becomes urgent.

Thank you.

[Wages_of_sin]

Hi, just a question. The countries GMC suspended on your account were getting the feed from the new domain, or the old one? In my case, I’ve been suspended only from Australia, as the overseas Spreadshop works only on the new domain, but the feed still doesn’t work.

I still get the old domain’s feed working on european Spreadshop, but USA, Canada, and Australian feeds get 100% items disapproved, and suspension on Australia.

[Britta_Spreadshop]

Hi @Adopteunlook,
can you give me a little more detail here? It’s had to figure out the problem from afar.
• which feed URLs are used
• which website is used to verify the shop
• what is the set up of these countries
Also: have you checked with google yet? Thanks!

[Wages_of_sin]

If that can help, the feed I was using on GMC that got my Australian account suspended is wages-of-sin

On overseas Spreadshop the old domain doesn’t work at all any more, and the new domain still doesn’t have a working feed (GMC sees the products, but they are all disapproved). The suspension reason is the same as reported by Adopteunlook: “Misrepresentation”.

On european Spreadshop the feed still only works with the old domain, and I’m getting impressions and some clicks. Also the domain used to verify my GMC accounts is the old one, of course.

I have removed all 3 overseas Spreadshop feeds to avoid any more suspensions.

[Adopteunlook]

The name of my shop: Adopteunlook
The url used: https://adopteunlook.myspreadshop.fr/

I have been using for many months a google merchant center and google ads account on the domain shop.spreadshirt.fr/adopteunlook with my own feed (I do not use the feed provided by Spreadshop which is incomplete, it has a limitation of the number of products). I had no problem.

Following your announcements, I made the changes on google analytics, google search console and on the same google merchant center account. I changed the URL to put the new one (so I didn’t create a new merchant account). I also modified my product feed to have the new url.

Since these changes my products are refused.

There seems to be a problem with the image crawling.

I crawl the image on this url. So far it has been working well.

https://image.spreadshirtmedia.net/image-server/v1/products/T752A2PA3739PT17X60Y15D187651502FS3258/views/1,width=1000,height=1000,appearanceId=2,crop=list,modelId=2024/jpeux-pas-jai-judo-debardeur-ultra-leger-femme.jpg

Do I have to start from scratch with a new merchant account ? How do others not have a problem with the new url ?

Tanks.

[Adopteunlook]

Hello,

I have the same feed for the 3 countries (a google sheet file).

I simply modified the urls to put the new one. Then I also changed the url in the account settings.

I specify that I am not a beginner. I have been using google merchant center and google ads for several years.

Do you have the “google-image” error like me?

[Wages_of_sin]

I was getting the robots.txt error when trying to use Spreadshop’s feed with the new domain. Using the old domain works ok.

My issue is just with overseas Spreadshop sites (USA, Canada, and Australia), where the old domain doesn’t work any more, so using the new domain still gets that robots.txt error, and my australian GMC account was suspended.

I assume that, if you modify your feed URL using the old domain, it should solve the error as it did for me. But consider that I’m using the original Spreadshop feed, not a custom one.

[Adopteunlook]

But in the account settings (business information, website), you left the old url ?
Or do you have the new url and a feed with the old url ?

[Adopteunlook]

@Britta_Spreadshop @Wages_of_sin

There is no doubt that the problem is with the robot.txt file.

I just did a test on the google tool :

https://search.google.com/test/rich-results

With a product url from my shop :

Jpeux pas jai judo - Débardeur ultra-léger Femme | Adopteunlook | Vêtements & Accessoires humoristiques

Results :

Your team should fix this for new urls.

Thanks.

[Wages_of_sin]

I never changed my Spreadshop URL. Spreadshirt recently changed the domain from shop.spreadshirt./SHOPNAME to SHOPNAME.myspreadshop. (new domain).

In GMC I initially tried changing to the new domain, and I noticed that nothing was being crawled due to robots.txt. So, I reverted to the old domain, and everything works well. I was informed by Spreadshirt staff that during this transition period the old domain will work this way. When the new domain will be fully implemented, the feed on it will work as it’s working now on the old domain.

For now, the only way to make the feed work is using the old domain in GMC, settings included.

[Adopteunlook]

I understand.

For information I opened the feed provided by Spreadshop.
There is in this file the new url …

So if I’m a beginner and use the Spreadshop feed, it’s not going to work!

There is currently no logic, Spreadshop must completely switch to the new URL.

[smallthings.ch]

I think one of the problems is also that the image is not on the same server as the shop itself (images.spreadshirtmedia.net).
Also if you use the shop on your own domain this also generates problems if google gets forwarded from shopname.myspreadshop.com to shopname.com for example.

[Wages_of_sin]

Hello, today I’ve got another “issue”. It seems that Spreadshirt’s shop team is claiming one of my GMC accounts. Come on, really?

Seriously, is this another “temporary issue”, or should I just reclaim it by myself?

[Adopteunlook]

So far this has never been a problem. This is only since the arrival of the new URL.
The problem comes from the robot.txt file, it is google itself which says it.
Spreadshirt needs to act fast to fix this. @Britta_Spreadshop
Thanks.