I will explain the scenario in detail,
there is on function which uses io.Pipe and returns a io.Reader with some data .
what i want to do is pass that data directly from that reader to my put request.
but when i pass directly to my api it give error : provided data is not matching with expected data because there is mismatch of offset value from where it is supposed to read.
My goal is to acheive this without storing the data in the memory.
URL which i am using to upload my content is a aws pre-signed url.
func ParallelPutRequest(chunkObject ChunkUrl, assetPath string, accessToken string, i int, bar *progressbar.ProgressBar, subtitleRelativeMap *sync.Map) error {
var dataReader io.Reader
if strings.Contains(assetPath, ".tar") {
path := strings.TrimSuffix(assetPath, ".tar")
dataReader, err := TraverseDirectory(path, subtitleRelativeMap)
fmt.Println("reader", dataReader)
//data,_=io.ReadAll(dataReader)
//if err != nil {
// return err
//}
} else {
file, err := os.Open(assetPath)
if err != nil {
return err
}
dataReader := io.NewSectionReader(file, chunkObject.Offset, chunkObject.Size)
//data, err = //io.ReadAll(dataReader)
//if err != nil {
//return err
//}
defer file.Close()
}
client := &http.Client{
Timeout: 1 * time.Minute,
Transport: &http.Transport{
DisableKeepAlives: true,
},
}
for attempt := 0; attempt < 4; attempt++ {
// seeker, ok := dataReader.(io.Seeker)
// if ok {
// _, err = seeker.Seek(0, io.SeekStart)
// if err != nil {
// return err
// }
// }
//reader:=bytes.NewReader(data)
//fmt.Println("Reader inside request", reader)
req, err := http.NewRequest("PUT", chunkObject.UploadUrl, dataReader)
if err != nil {
return err
}
req.Header.Add("Content-Type", "application/octet-stream")
resp, err := client.Do(req)
if err != nil {
if err, ok := err.(net.Error); ok && err.Timeout() {
if attempt == 3 {
return err
}
continue
} else if _, ok := err.(*url.Error); ok {
if attempt == 3 {
return err
}
continue
} else {
slog.Error("upload.chunk.error", "attempt", attempt, "err", chunkObject.UploadUrl)
return err
}
}
responsbody, _ := io.ReadAll(resp.Body)
defer resp.Body.Close()
if err == nil && resp.StatusCode == http.StatusOK {
mutex.Lock()
bar.Add(int(chunkObject.Size))
mutex.Unlock()
return nil
}
slog.Error("upload.chunk.error", "attempt", attempt, "response", string(responsbody))
time.Sleep(1 * time.Second)
}
return nil
}
In the above code traverdirectory function returns a io.Reader which i want to pass it on my put request and also in else condition io.NewSectionReader returns io.SectionReader
(when i do io.ReadAll (that converts all the data into byte slice and add it into bytes.NewReader() it works totally fine, concern is it will consume memory)
help will be appricieated.
